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The  Department  of  Defense  (DoD)  High  Performance  Computing  Modernization  Program 
(HPCMP)  provides  world-class,  commercial,  high-end  high  performance  computing  capabilities 
to  the  DoD  science  and  technology  and  test  and  evaluation  communities.  The  program  supports 
over  600  high  performance  computing  projects.  The  user  base  exceeds  4,000  scientists  and 
engineers,  who  are  located  at  more  than  100  DoD  laboratories,  test  centers,  contractor,  and 
academic  sites.  In  an  era  of  Homeland  Security  and  the  War  on  Terrorism,  the  contributions 
made  by  the  HPCMP  community  are  of  increasing  importance,  providing  the  answers  to 
tomorrow’s  questions. 

These  success  stories  focus  on  the  contributions  made  by  the  HPC  community  to  the 
transformational  goals  of  the  Department.  Whether  through  long-term  research  or  more 
immediate  support  to  address  critical  operational  issues,  our  scientists  and  engineers  are  now 
routinely  using  HPCMP  resources  in  a  pervasive  environment  to  assist  in  the  transformation  of 
our  military  forces.  They  are  assisting  in  the  protection  of  critical  bases  of  operations  by 
simulating  chemical  warfare  agents  and  their  antidotes,  as  well  as  blast  protection  from 
explosions.  Others  are  conducting  research  that  will  ensure  the  integrity  of  our  information 
systems  in  the  face  of  attack.  HPCMP  supported  research  will  allow  the  projection  and 
sustainment  of  US  forces  in  distant  hostile  environments  through  better  weapons  system  design. 
DoD  scientists  are  using  HPCMP  resources  to  improve  synchronization  within  the  battlespace 
environment  and  to  develop  the  next  generation  of  tools  needed  to  deny  enemies  sanctuary.  Still 
others  are  developing  new  materials  to  enhance  the  capability  and  survivability  of  space  systems, 
leveraging  information  technology  and  innovative  concepts  to  develop  joint  operational  pictures. 
Enhancing  our  response  to  terrorist  threats,  improving  the  design  of  structures  and  weapons 
systems,  conducting  robust  testing  programs,  and  comprehensively  evaluating  the  structural 
design  of  entire  platforms  -  all  are  represented  in  this  technical  report,  and  all  require  high 
performance  computing. 

The  threat  environment  of  the  21sl  century  requires  a  defense  posture  that  is  flexible, 
innovative,  and  fully  capable  of  superior  response  across  the  spectrum  of  conflict.  The 
Department  of  Defense’s  scientific,  engineering,  and  testing  communities  are  fulfilling  those 
challenges  using  HPCMP  resources. 
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I NTRODUCTION 


Introduction 


The  Department  of  Defense  (DoD)  High  Performance  Computing  Modernization  Program 
(HPCMP)  was  initiated  in  1992  by  a  Congressional  directive  to  modernize  the  DoD  laboratories 
high  performance  computing  (HPC)  capabilities.  The  HPCMP  operates  under  the  auspices  of  the 
Deputy  Under  Secretary  of  Defense  for  Science  and  Technology.  Since  its  inception,  the  program 
has  realized  great  success  in  its  efforts  to  modernize  the  Department’s  laboratories  and  test 
centers  to  levels  on  par  with  other  world-class  organizations. 

This  technical  report,  HPCMP  Contributions  to  DoD  Mission  Success  2002,  will  provide  insights 
into  how  the  program  has  evolved  and  chronicle  some  of  the  near  term  and  long  term  national 
defense  work  done  by  the  science  and  technology  (S&T)  and  test  and  evaluation  (T&E) 
communities  using  HPCMP  resources.  The  High  Performance  Computing  Modernization  Program 
is  placing  critical  enabling  21 st  century  technology  into  the  hands  of  the  Department’s  science, 
technology,  test,  and  evaluation  communities  that  allows  them  to  realize  the  Department’s 
transformational  goals. 

This  report  is  organized  into  four  sections.  The  first  section  provides  an  overview  of  the  program 
and  its  accomplishments  as  well  as  an  explanation  of  how  the  stories  are  grouped.  The  second 
section  consists  of  a  selection  of  stories  detailing  the  scientific  research  and  engineering  work 
done  by  the  scientists  and  engineers  in  the  Department’s  laboratories  and  test  centers,  working 
with  academia  and  industry  in  seamless  high  performance  computing  environments.  The  third 
section  provides  a  detailed  discussion  of  the  structure  of  the  HPCMP  and  the  fourth  section  is  a 
list  of  acronyms  used  throughout  the  publication. 


Overview  of  the  High  Performance  Computing 
Modernization  Program 


The  Department  of  Defense  High  Performance  Computing  Modernization  Program  was 
established  as  a  Joint  Program  Office  in  1993.  This  mandate  was  subsequently  expanded  to 
include  the  Department’s  test  and  evaluation  community  as  well  as  the  Missile  Defense  Agency 
(MDA)  [formerly  the  Ballistic  Missile  Defense  Organization],  In  the  following  years,  the  Program 
has  taken  fragmented,  Service-oriented  research  and  engineering  efforts  and  transformed  them 
into  a  nearly  seamless,  joint  operating  environment  that  leverages  the  best  practices  of  all  of  the 
Services  and  Agencies  as  well  as  the  national  academic  and  industrial  infrastructure. 

Mission  and  Vision 

The  High  Performance  Computing  Modernization  Program’s  mission  is  to 

“Deliver  world-class,  commercial,  high-end,  high  performance  computational 
capability  to  the  DoD’s  science  and  technology  (S&T)  and  test  and  evaluation 
(T&E)  communities,  facilitating  the  rapid  application  of  advanced  technology 
into  superior  warfighting  capabilities.  ” 
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-  Introduction  - 

The  Director  of  the  HPCMP  has  articulated  that  mission  in  the  following  vision  for  the  Program: 

[To  create]  “A  pervasive  culture  existing  among  DoD’s  scientists  and 
engineers  where  they  routinely  use  advanced  computational  environments  to 
solve  the  most  demanding  problems.  ” 

Today’s  HPC  Modernization  Program  supports  over  4,000  scientists  and  engineers  working 
through  resources  located  at  sites  throughout  the  United  States,  including  Alaska  and  Hawaii.  As 
Figure  1  illustrates,  the  Program  has  provided  a  phenomenal  increase  in  the  computational 
power  available  to  the  community,  along  with  the  tools  and  intellectual  expertise  needed  to  fully 
use  the  resources  effectively.  The  innovative  and  groundbreaking  work  being  done  using 
program  resources  is  found  throughout  the  Program’s  components  —  of  the  current  41  DoD 
Challenge  Projects  (listed  in  Section  3),  over  90%  are  directly  or  indirectly  related  to  one  or  more 
transformational  objectives  (see  page  7).  The  software  applications  support  initiative  contributes 
daily  to  the  expanding  software  suite  available  to  the  scientific  and  engineering  efforts.  The 
Defense  Research  and  Engineering  Network  (DREN)  is  a  premier  departmental  example  of 
leveraging  information  technology  as  well  as  providing  groundbreaking  defensive  input/output  (1/ 
O)  through  its  work  with  network  intrusion  devices  and  multi-tiered  access. 


HPCMP  Organization 


The  HPCMP  has  three  components,  which  are  detailed  in  Section  3.  These  components  -  HPC 
Centers,  Networking,  and  Software  Applications  Support  —  provide  a  technologically  advanced 
computational  environment  to  support  the  ongoing  and  emerging  needs  of  the  Department’s 
laboratories  and  test  centers.  The  High  Performance  Computing  Centers,  shown  in  Figure  2, 
provide  local  and 
generalized  support  to  the 
HPCMP  community  based 
on  Service  and 
Agency-validated 
requirements  and  priorities. 

The  centers  also  support 
outreach  to  the 
communities  in  which  they 
reside,  encouraging 
students  to  become 
involved  with  engineering 
and  scientific  disciplines. 
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Figure  1.  DoD  S&T  and  T&E  growth  in  computational  capability 


The  HPCMP  community 
consists  of  over  4,000 
scientists,  engineers, 

computer  specialists,  networking  specialists,  and  security  experts  working  throughout  the  United 
States.  All  three  Services  and  several  Defense  Agencies  participate  in  the  program. 
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2002  Success  Stories 


Figure  2.  HPC  Centers  (As  of  December  2002) 


As  shown  in  Figure  3,  the  user  base  is  diverse,  drawing  from  the  government  workforce, 
academia,  and  industry. 
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*  Contractor  includes  industrial  and  academic  partners 

Figure  3.  User  Demographics 

The  community  is  organized  around  10  computational  scientific  disciplines,  which  are  known  as 
Computational  Technology  Areas  (CTA).  The  CTAs  are  described  in  Table  1 .  A  nationally 
recognized  expert  in  that  discipline  leads  each  CTA.  In  addition,  there  are  a  subset  of  projects 
which  focus  on  the  Department’s  highest  priority,  most  computationally  demanding  work  —  the 
DoD  Challenge  Projects. 


—  Introduction 


Table  1 .  Computational  Technology  Areas 


CTA 

Description 

CSM 

Computational  Structural 
Mechanics 

Covers  the  high  resolution,  multi-dimensional  modeling  of  materials  and 
structures  subjected  to  a  broad  range  of  loading  conditions  including 
static,  dynamic,  and  impulsive.  CSM  encompasses  a  wide  range  of 
engineering  problems  in  solid  mechanics  such  as  linear  elastic  stress 
analysis.  CSM  is  used  for  basic  studies  in  continuum  mechanics,  stress 
analysis  for  engineering  design  studies,  and  structural  and  material 
response  predictions  to  impulsive  loads. 

CFD 

Computational  Fluid 

Dynamics 

Covers  high  performance  computations  whose  goal  is  the  accurate 
numerical  solution  of  the  equations  describing  fluid  and  gas  motion.  CFD 
is  used  for  basic  studies  of  fluid  dynamics  for  engineering  design  of 
complex  flow  configurations  and  for  predicting  the  interactions  of 
chemistry  with  fluid  flow  for  combustion  and  propulsion. 

CCM 

Computational  Chemistry 
and  Materials  Science 

Covers  the  computational  research  tools  used  to  predict  basic  properties 
of  new  chemical  species  and  materials  which  may  be  difficult  or 
impossible  to  obtain  experimentally.  Within  DoD,  quantum  chemistry  and 
molecular  dynamics  methods  are  used  to  design  new  chemical  systems 
and  solid  state  modeling  techniques  are  employed  in  the  development  of 
new  high-performance  materials. 

CEA 

Computational 
Electromagnetics  and 
Acoustics 

Covers  the  high-resolution,  multi-dimensional  solutions  of  Maxwell's 
equations  as  well  as  the  high-resolution,  multi-dimensional  solutions  of 
the  acoustic  wave  equations  in  solids,  fluids,  and  gases.  Some  DoD 
applications  include  calculating  the  electromagnetic  fields  about  antenna 
arrays,  the  electromagnetic  signatures  of  tactical  ground,  air,  sea  and 
space  vehicles,  the  electromagnetic  signature  of  buried  munitions,  high 
power  microwave  performance,  as  well  as  the  interdisciplinary 
applications  in  magnetohydrodynamics  and  laser  systems. 

CWO 

Climate/Weather/Ocean 
Modeling  and  Simulation 

Is  concerned  with  the  accurate  numerical  simulation  of  the  earth's  climate 
including  the  simulation  and  forecast  of  atmospheric  variability  and 
oceanic  variability.  CWO  includes  the  development  of  numerical 
algorithms  and  techniques  for  the  assimilation  of  in-situ  and  remotely 
sensed  observations  into  numerical  prediction  systems. 

SIP 

Signal/lmage  Processing 

Emphasizes  research,  evaluation,  and  test  of  the  latest  signal  processing 
concepts  directed  toward  these  embedded  systems.  This  will  enable  the 
traditional  expensive,  military-unique  'black  boxes'  required  to  implement 
high-speed  signal/image  processing  to  be  replaced  by  commercial-off- 
the-shelf  HPC-based  equipment. 

FMS/ 

C4I 

Forces  Modeling  and 
Simulation 

Focuses  on  force  level  modeling  and  simulation  for  training,  analysis,  and 
acquisition.  The  acquisition  domain  includes  research  and  development, 
test  and  evaluation,  and  production  and  logistics.  The  overarching 
Simulation  Based  Acquisition  domain  is  included. 

EQM 

Environmental  Quality 
Modeling  and  Simulation 

Covers  the  high-resolution,  three-dimensional  Navier-Stokes  modeling  of 
hydrodynamics  and  contaminant  and  multi-constituent  fate/transport 
through  the  aquatic  and  terrestrial  ecosystem  and  wetland  subsystems, 
their  coupled  hydrogeologic  pathways,  and  their  interconnections  with 
numerous  biological  species. 

CEN 

Computational  Electronics 
and  Nanoelectronics 

Uses  advanced  computational  methods  to  model  and  simulate  complex 
electronics  for  communications,  command,  control,  electronic  warfare, 
signal  intelligence,  sensing,  and  related  applications.  Use  of  high 
performance  computing  assets  enables  the  DoD  electronics  community  to 
solve  complex  problems,  explore  new  concepts,  gain  insight  and 
improved  understanding  of  the  underlying  physics,  perform  virtual 
prototyping,  and  test  new  ideas. 

IMT 

Integrated  Modeling  and  Test 
Environments 

Addresses  the  application  of  integrated  models,  simulation  tools  and 
techniques  with  live  tests  and  hardware-in-the-loop  simulations.  IMT  also 
provides  proof-of-concept  by  using  technology-based  war  gaming 
modeling  and  software  integration  tools  for  the  simulation  of  weapon 
component  subsystems  and  systems  in  a  virtual  operational  context. 
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Within  the  DoD  Challenge  Projects,  scientists  and  engineers  are  using  program  resources  to  work 
on  projects  such  as  the  evaluation  and  retrofit  for  blast  protection  in  urban  terrain;  three- 
dimensional  modeling  and  simulation  of  bomb  effects  for  obstacle  clearance;  the  active  control  of 
fuel  injectors  in  full  gas  turbine  engines;  and  the  seismic  signature  simulations  for  tactical  ground 
sensor  systems  and  underground  facilities.  Others  are  working  on  improving  naval  capabilities 
through  unprecedented  modeling  of  the  climate,  oceans,  and  weather  conditions  of  the  world  as 
well  as  investigating  littoral  environments.  Still  others  directly  support  the  airborne  laser  and 
missile  defense  programs.  Program  resources  helped  to  develop  thermobaric  weapons  and  to 
provide  forensic  data  for  investigation  of  terrorist  bombings  and  consequence  management. 
Additionally,  program  scientists  and  engineers  are  developing  codes  to  help  development  of  new 
materials  that  will  enhance  our  capabilities  and  survivability  in  space. 


HPCMP  Support  of  Transformation  and  the  War  on 
Terrorism 


Transformation  Operational  Goals 

The  stories  included  in  this  Technical  Report  are  grouped  to  reflect  the  Department’s 
transformation  operational  goals.  The  2001  Quadrennial  Defense  Review  Report  states  that  “the 
purpose  of  transformation  is  to  maintain  or  improve  US  military  preeminence  in  the  face  of 
potential  disproportionate  discontinuous  changes  in  the  strategic  environment.”  It  also  states  that 
“the  transformation  must  therefore  be  focused  on  emerging  strategic  and  operational  challenges 
and  the  opportunities  created  by  these  challenges.”  The  2001  Quadrennial  Defense  Review 
Report  lists  the  six  critical  operational  goals  that  provide  the  focus  for  DoD’s  transformation 
efforts:1 

1 .  Protecting  critical  bases  of  operations  (US  homeland,  forces  abroad,  allies,  and  friends)  and 
defeating  chemical,  biological,  radiological,  nuclear,  or  high-yield  explosive  weapons  and  their 
means  of  delivery; 

2.  Assuring  information  systems  in  the  face  of  attack  and  conducting  effective  information 
operations; 

3.  Projecting  and  sustaining  US  forces  in  distant  anti-access  or  area-denial  environments  and 
defeating  anti-access  and  area-denial  threats; 

4.  Denying  enemies  sanctuary  by  providing  persistent  surveillance,  tracking,  and  rapid 
engagement  with  high-volume  precision  strike,  through  a  combination  of  complementary  air 
and  ground  capabilities,  against  critical  mobile  and  fixed  targets  at  various  ranges  and  in  all 
weather  and  terrains; 

5.  Enhancing  the  capability  and  survivability  of  space  systems  and  supporting  infrastructure;  and 

6.  Leveraging  information  technology  and  innovative  concepts  to  develop  an  interoperable,  joint 
Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance  architecture  and  capability  that  includes  a  tailorable  joint  operational  picture. 


1  2001  Quadrennial  Defense  Review  Report,  September  30,  2001,  pg.  30;  www.defenselink.mil/pubs/qdr2001.pdf 


“Our  challenge  in  the  21st  century  is  to  defend  our  cities  and  our  infrastructure  from  new 
forms  of  attack  while  protecting  our  forces  over  long  distances  to  fight  new  adversaries” 

—  Defense  Secretary  Donald  Rumsfeld  in  a  speech  to  students  at  the 
National  Defense  University,  January  21,  2002 


Protect  Bases 
of  Operation 


Chemical  Defense 
Combating  Terrorism 
Consequence  Management 


Missile  Defense 
Biological  Defense 
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References  (from  top  to  bottom): 

Defense  Against  Chemical  Warfare  Agents  by  Margaret  Hurley,  Jeffery  Wright,  William  White,  and  Gerald  Lushington 

Overlaid  crystal  structures  of  Torpedo  californica  AChE  with  reversible  (red)  and  irreversible  (blue)  inhibitors  bound  to  active  site 
residues  (green) 

Effect  of  Planar  and  Grid  Tail  Fins  on  Canard  Roll  Control  Effectiveness  on  a  Generic  Missile  by  James  DeSpirito 

Canard-controlled  missile  with  grid  fins 

Dispersion  of  Biological  or  Chemical  Aerosols  in  Cities  by  Shahrouz  Aliabade,  Robert  Bafour,  Jalal  Abedi,  and  Andrew 
Johnson 

Geometric  model  of  downtown  Atlanta,  GA,  to  be  used  for  fine-scale  (1  to  5  meter)  numerical  simulations  of  biological/chemical 
dispersion 

Modeling  Structural  Response  to  Blasts  by  Joseph  Baum 

Airblast/fragments  impact  on  a  steel  plate 
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2002  Success  Stories 


Since  the  Cuban  missile  crisis  in  1962,  no  single  event  has  shocked  the  nation  into  the  reality  of  our 
vulnerability  as  did  the  events  of  September  1 1 , 2001 .  The  need  to  defend  the  US  homeland  and 
overseas  bases  has  spurred  increased  attention  to  developing  defenses  against  chemical  warfare 
agents,  cleaning  up  contaminated  soils  and  groundwater,  strengthening  the  resistance  of  buildings  to 
blast  events,  and  developing  technologies  to  detect  potential  terrorist  attacks  from  land  or  sea.  The 
following  success  stories  share  a  common  theme  —  the  unprecedented  challenge  of  protecting  our 
infrastructure  and  warfighters  by  taking  advantage  of  the  invaluable  HPC  resources  made  available 
through  the  High  Performance  Computing  Modernization  Program. 

The  threat  of  nerve  and  other  chemical  warfare  agents,  a  major  concern,  has  been  the  focus  of  studies 
such  as  those  by  Margaret  Hurley  and  her  associates  (“Defense  Against  Chemical  Warfare  Agents”). 

In  this  investigation,  HPC  resources  are  being  used  to  model  the  interaction  of  nerve  agents  and  the 
enzyme  acetylcholinesterase  (AchE),  enabling  numerous  advancements  over  previous  simulations 
based  on  classical  molecular  dynamics  techniques.  In  another  research  effort,  Robert  Bernard  et  al. 
(“Pore-Scale  Contaminant  Transport”)  describe  improved  measurement  and  modeling  systems  for  the 
study  of  chemical  and  biological  agent  transport  in  permeable  media.  Advances  in  HPC  technologies 
have  made  possible  the  complex  computations  necessary  for  these  improved  systems  and 
subsequent  successful  applications  on  military  sites. 

Contamination  of  soils  and  groundwater  from  explosives,  fuels,  or  other  agents  can  be  hazardous  to 
the  environment  and  to  human  beings,  as  well  as  costly  to  clean  up.  Computational  simulation 
methods  were  employed  in  a  project  led  by  Jerry  Leszczynski  (“Exploring  Cleanup  Technologies  for 
Explosives  in  Soils  and  Groundwater”)  to  develop  remediation  technologies  for  soils  and  groundwater 
contaminated  by  explosives.  Using  a  quantum-chemical  (QC)  approach,  Dr.  Leszczynski’s  team 
investigated  the  breakdown  of  nitrobenzene  through  interaction  with  the  surface  of  an  iron  oxide  (FeO) 
mineral  normally  found  in  soil.  Parallel  supercomputing  resources  significantly  enhanced  the  study.  In 
another  contamination  study,  J.D.  Nieber  and  his  associates  (“Promoting  Environmental  Stewardship: 
Understanding  Fingering  Flow  in  Unsaturated  Porous  Media”)  focused  on  the  movement  of  fluids  in 
porous  media  in  order  to  determine  effective  remediation  processes.  Dr.  Nieber’s  team  used  a  new 
approach  to  solve  the  multi-phase  fluid  problem. 

Simulations  on  high  performance  computers  are  invaluable  in  the  development  of  technologies  to 
protect  the  nation’s  infrastructure  from  blasts  or  explosives.  Joseph  Baum  (“Modeling  Structural 
Response  to  Blasts”)  describes  the  use  of  HPC  and  a  coupled  Computational  Fluid  Dynamics  (CFD)/ 
Computational  Structural  Dynamics  (CSD)  code  to  model  the  processes  involved  in  a  blast  event,  and 
specifically  the  interactions  of  the  blast  and  the  structural  response.  Starting  with  a  “glued-mesh” 
approach,  Dr.  Baum  and  his  team  of  researchers  have  developed  a  more  effective  “embedded-mesh” 
technique.  The  computational  simulations  using  this  approach  illustrate  the  structural  responses  to  be 
expected  in  a  car  or  truck  bombing.  These  findings  will  enable  the  design  of  structures  that  are  more 
resistant  to  blast  events.  Tommy  Bevins  et  al.  (“Evaluation  of  Blast  Response  to  the  Pentagon”) 
provide  an  in-depth  description  of  the  retrofit  designs  for  repair  of  the  Pentagon  after  the  September 
1 1 , 2001 ,  attack.  A  number  of  factors  had  to  be  taken  into  consideration,  including  the  loads  on  the 
components  of  the  structure,  the  response  of  these  components  to  the  loads,  and  improved  safety  for 
the  occupants.  The  complex  computations  required  millions  of  equations  and  thousands  of  time 
steps. 

Defense  measures  include  protection  and  surveillance  devices  that  are  essential  to  homeland  security. 
Modeling  approaches  being  developed  by  John  Starkenberg  and  Toni  Dorsi  (“Modeling  Methods  for 
Assessment  of  Gun-Propellant  Vulnerability”)  will  be  used  to  analyze  the  reactive  responses  of 
propellant  beds  to  unplanned  stimuli.  Simulation  results  will  be  used  in  the  design  of  propellant  beds 
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with  the  goal  of  reducing  the  violence  of  these  reactive  responses.  The  modeling  approaches 
developed  in  this  study  are  being  used  for  the  Army’s  Future  Combat  System  in  the  design  of  high- 
loading-density  gun-propellant  charges.  In  another  research  effort,  infrared  search  and  track  systems 
under  development  offer  improved  surface  and  air  defense  capabilities  (“A  Highly  Parallel,  Scalable 
Environment  for  Developing  Search  and  Track  Algorithms,”  Cottel,  Douma,  and  Henderson).  These 
systems  require  highly  computationally  intensive  spatial,  spectral,  and  temporal  processing  for  real¬ 
time  applications.  Jeffery  Dugan  and  his  associates  (“Robust  Scalable  Information  Processing  for 
Multistatic  Sonars”)  describe  improved  algorithms  that  increase  the  performance  of  acoustic  sonar, 
enhancing  the  ability  to  detect  submarine  threats  and  reducing  false  pings.  HPC  resources  were  used 
to  test  and  validate  the  algorithm  for  an  improved  range  rate  estimator  for  the  Hyperbolic  Frequency 
Modulated  (HFM)  waveform,  thus  reducing  the  percentage  of  high-speed  clutter  as  well  as  the 
algorithmic  and  operator  resources. 

The  significance  of  these  projects  and  their  reliance  on  HPC  technologies  cannot  be  underestimated. 
The  large  storage  and  processing  capabilities  made  possible  by  HPC  resources  have  enabled  the 
robust  testing,  accuracy,  and  time  and  cost  effectiveness  that  are  essential  to  homeland  defense. 


Lee  Byrne 

US  Army  Corps  of  Engineers,  Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 

Dr,  Jeffery  P,  Holland 

US  Army  Corps  of  Engineers,  Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 
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Defense  Against  Chemical  Warfare  Agents 

Margaret  Hurley 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 
Jeffery  Wright 

Soldier  and  Biological  Chemical  Command  (SBCCOM),  Natick,  MA 
William  White 

Edgewood  Chemical  and  Biological  Center  (ECBC),  Aberdeen  Proving  Ground,  MD 

Gerald  Lushington 

University  of  Kansas,  Lawrence,  KS 


The  threat  of  nerve  agents  and  other  chemical 
warfare  agents  has  been  a  constant  specter 
over  the  last  century,  and  will  continue  to  be 
so.  Despite  international  conventions 
banning  their  use,  confirmed  or  alleged  incidents 
have  occurred  over  the  last  two  decades  alone  in 
Afghanistan,  Iraq,  Iran,  and  Japan.  Defense  against 
and  therapy  for  exposure  to  chemical  warfare  agents 
will  remain  a  cornerstone  of  protection  for  the 
warfighter  (as  well  as  civilians)  in  years  to  come. 

Investigating  Nerve  Agents 

While  much  information  has  been  obtained  on  the 
function  of  nerve  agents  and  one  of  their  primary 
targets,  the  enzyme  acetylcholinesterase  (AChE), 
substantial  questions  remain.  Researchers  continue 
to  investigate  the  mechanism  of  their  interaction,  as 
well  as  the  mechanism  of  related  compounds  used 
for  therapy  and  prophylaxis.  Experimental  studies  on 
these  systems  are,  by  their  very  nature,  dangerous 
and  expensive,  making  this  an  ideal  target  for 
theoretical  investigation.  Dr.  Margaret  Hurley  and  her 
associates  are  attempting  to  understand  the  key 
molecular  features  leading  to  reversible  and 
irreversible  binding  in  the  active  site  of 
acetylcholinesterase  (and  related  reactions).  In 
doing  so,  they  will  develop  a  solid  basis  for  devising 
chemical  prophylaxis  and  therapeutics  techniques, 
as  well  as  developing  a  reliable  technique  for 
prediction  of  new,  possibly  dangerous,  compounds. 


Understanding  the  Chemistry 

Previous  simulations  of  these  systems  have  been 
performed  by  using  classical  molecular  dynamics 
techniques,  or  by  quantum  mechanical  study  of  a 
severely  truncated  section  of  the  active  site  residues. 
Classical  techniques  are  extremely  useful  tools  in 
understanding  gross  structural 
features  contributing  to  the 
active  site  chemistry,  but 
cannot  provide  the  detailed 
energetics  necessary  to 
understanding  the  chemistry 
involved.  Truncated  models 
may  leave  out  surrounding 
residues  and  other  features  that 
play  a  key  role  in  determining 
the  mechanism.  Mixed 
quantum/classical  (QM/MM) 
methods  have  been  generally 
accepted  as  a  reasonable  compromise,  capable  of 
simultaneous  representation  of  both  the  subtle 
reactive  (QM)  phenomena  and  the  bulk  (MM) 
environment  dependence. 

Applying  HPC  to  Higher  QM/MM 
Models 

Access  to  HPC  resources  (both  hardware  and 
software)  has  allowed  the  research  community  to 
make  numerous  improvements  over  previous 
treatments  of  AChE  systems.  They  are  able  to  treat 
substantially  larger  and/or  higher  accuracy  QM 
models,  and  to  run  larger  and/or  higher  accuracy 


“Experimental  studies 
on  these  systems  are, 
by  their  very  nature, 
dangerous  and 
expensive,  making  this 
an  ideal  target  for 
theoretical 
investigation." 
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Figure  1 .  Overlaid  crystal  structures  of  Torpedo  californica  AChE  with 
reversible  (red)  and  irreversible  (blue)  inhibitors  bound  to  active  site 
residues  (green) 


References 

Hurley,  M.M.,  Wright,  J.B.,  Lushington,  G.L.,  and  White, 
W.E.,  “QM  and  Mixed  QM/MM  Simulations  of  Model  Nerve 
Agents  with  Acetylcholinesterase,”  Theoretical  Chemistry 
Accounts  (Special  Issue  on  QM/MM),  2002.  [Accepted] 

Hurley,  M.M., Wright,  J.B.,  Lushington,  G.L.,  and  White, 
W.E.,  Studies  of  Acetylcholinesterase  and  Model  Nerve 
Agent  Interactions:  a  QM/MM  Approach.  Presented  at  the 
10th  Conference  on  Current  Trends  in  Computational 
Chemistry,  Jackson,  MS,  November  1-3,  2001. 

CTA:  Computational  Chemistry  and  Materials  Science 
Computer  Resources:  SUN  El  0000  and  SGI  Origin 
3000  [ARL],  IBM  SP  P3  [ASC],  and  Cray  T3E  [ERDC] 


QM/MM  models.  High  performance  computing 
resources  supported  one  of  the  first  systematic 
comparisons  of  competing  QM/MM  methodologies  to 
determine  the  effect  of  the  algorithm  on  results.  The 
program  also  provided  sufficient  resources  to 
perform  a  mapping  of  model  details  (size,  method, 
level  of  accuracy,  etc.)  and  their  effect  on  results. 

Reproducing  the  Short  Strong 

Hydrogen  Bond 

The  results  to  date  have  demonstrated  the 
effectiveness  of  these  models.  The  research  team 
has  been  able  to  reproduce  the  Short  Strong 
Hydrogen  Bond  (SSHB)  in  the  active  site  residues 
seen  experimentally  in  the  bare  AChE  enzyme.  In 
addition,  the  models  confirm  greatly  reduced  energy 
barriers  to  the  phosphylation  reaction  in  enzyme 
relative  to  related  calculations  of  phosphonate 
hydrolysis  in  aqueous  solution.  They  have  also 
demonstrated  the  charge  transfer  between  agent  and 
oxyanion  hole  residues  of  enzyme,  thus  showing 
importance  of  quantum  effects  in  non-active  site 
residues  to  the  reaction. 

Currently,  the  team  is  conducting  a  detailed  mapping 
of  the  aging  reaction  (by  which  the  agent  becomes 
irreversibly  bound  to  the  enzyme)  and  the  origin  of 
stereospecificity  in  these  systems,  as  well  as 
implementation  of  new  higher  accuracy  QM/MM 
methods. 
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Pore-Scale  Contaminant  Transport 

Robert  Bernard,  Stacy  Howington,  and  John  Peters 
Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 

Robert  Maier 

Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 
Daniel  Kroll  and  H.  Ted  Davis 
University  of  Minnesota,  Minneapolis,  MN 


Modeling  gas  and  liquid  flows  in 
permeable  media  is  important  in 
understanding  the  dispersal  and 
persistence  of  chemical  and  biological 
agents.  Until  recently,  the  direct  numerical  simulation 
(DNS)  of  flow  and  transport  in  permeable  media  was 
regarded  as  a  computationally  intractable  problem. 
Resolving  flow  even  in  a  small  sample  requires  an 
enormous  computational  effort  due  to  the  complex 
physical  boundaries.  However,  recent  advances  in 
the  application  of  HPC  technologies  have 
demonstrated  that  such  computations  are  not  only 
feasible,  but  can  be  performed  on  a  reasonable 
production  schedule,  leading  to  improvements  in 
measurement  and  modeling  systems. 


Pore-Scale  Simulations 

DNS  in  permeable  media  is  often  referred  to  as  pore- 
scale  flow  and  transport  because  flow  is  resolved  on 
the  length  scale  of  typical  pore  spaces,  using  the 
three-dimensional  Navier-Stokes  equations.  Species 
transport  is  modeled  on  the  same  length  scale  by 
solving  a  convection-diffusion  equation,  either 
coupled  with  the  flow  calculation  or,  more  typically, 
using  a  steady-state  flow  field.  As  shown  in  Figure  1 , 
no  tunable  parameters  are  needed  to  solve  these 
equations;  only  the  molecular  viscosity  and  diffusion 
coefficients  need  be  specified. 


+  <v  .’Pjv 


=  pi’  —  Vp  +  p  V  ■  v 


3f 


Figure  1 .  Navier-Stokes  (top)  and  convection-diffusion  equations 
(bottom)  where  v  denotes  the  fluid  velocity,  p  the  pressure,  c  the 
species  concentration,  i  the  viscosity,  Dm  the  coefficient  of  molecular 
diffusion 


Early  examples  of  pore-scale  simulation  were 
conducted  for  civilian  projects,  including  research  on 
oil  recovery  conducted 
at  Los  Alamos  National 
Laboratory  (LANL)  in  the 
early  1990’s.  The 
technology  was 
subsequently  applied  to 
DoD  projects  within  the 
Environmental  Quality 
Modeling  and  Simulation 
(EQM)  Computational 
Technology  Area, 
including  modeling  of 
subsurface 

contamination  on  military 
sites.  These  simulations 
required  up  to  512 
processors,  200 
gigabytes  main  memory,  and  three  days  of  wall-clock 
processing  time. 


“Until  recently,  the  direct 
numerical  simulation  (DNS)  of 
flow  and  transport  in 
permeable  media  was 
regarded  as  a  computationally 
intractable  problem.  Recent 
advances  in  the  application  of 
HPC  technologies  have 
demonstrated  that  such 
computations  are  not  only 
feasible,  but  can  be  performed 
on  a  reasonable  production 
schedule,  leading  to 
improvements  in  measurement 
and  modeling  systems.” 


Modeling  Flow  Columns 

One  application  of  pore-scale  simulation  is  the 
modeling  of  physical  experiments  in  laboratory  flow 
columns.  This  type  of  simulation  has  revealed 
artifacts  inherent  in  experimental  measurements  and 
protocols.  Simulations  predicted  the  existence  of  a 
damping  effect  in  Nuclear  Magnetic  Resonance 
(NMR)  velocity  measurements  caused  by  molecular 
diffusion.  Simulations  also  predicted  that  in  flow 
column  experiments,  the  confining  wall  enhances 
experimental  measurements  of  the  hydrodynamic 
dispersion  coefficient  compared  to  an  unconfined 
permeable  medium  (Figure  2). 
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Figure  2.  An  example  of  permeable  media:  thin  section  of  a  cylinder 
(r  =  25d)  packed  with  monodisperse  spheres  of  diameter  d  (full 
cylinder  includes  over  one  million  spheres) 


Further  applications  of  the  pore-scale  simulation  have 
been  identified,  including  the  flow  of  combustion 
gases  in  ceramic  foams,  particle  residence  time  in 
chromatographic  columns  and  spectrographic 
sampling  chambers,  and  shear  stress  calculations  in 
saturated  soil  microstructures  (Figures  3-6).  A  new 
challenge  is  to  extend  the  technology  to  the  flow  of 
colloidal  suspensions,  including  aerosols. 


Figure  3.  Three-dimensional 
view  of  dissolved  chemical 
concentration  in  a 
polydisperse  packed  bed. 
Flow  is  from  top  to  bottom, 
dissolving  a  thin  chemical 
film  from  the  solid  surfaces. 
The  intensity  of  red  color 
denotes  the  relative 
concentration. 


Figure  4.  Quasi-2D  view 
(thin  section)  of  dissolved 
chemical  concentration  in 
polydisperse  bed 
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Figure  5.  Contours  of 
dissolved  chemical 
concentration  in  cross  section 
of  a  three-dimensional 
monodisperse  packed  bed. 
Flow  of  water  is  from  top  to 
bottom,  washing  a  thin 
chemical  film  from  solid 
surfaces.  Darker  contours 
denote  higher  concentration. 


Figure  6.  Residual  chemical 
film  on  solid  surfaces  after 
washing  is  highlighted  in 
black 
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Exploring  Cleanup  Technologies  for  Explosives  in 
Soils  and  Groundwater 

Jerzy  Leszczynski 

Jackson  State  University,  Jackson,  MS 

Sponsor:  Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 


Computational  simulation  methods  have 
become  powerful  research  tools  for 
developing  effective,  passive,  in  situ 
clean-up  technologies  for  explosives  in 
soils  and  groundwater  within  the  US  Army 
environmental  restoration  mission.  Among  these 
computational  methods,  the  quantum-chemical  (QC) 
approach  is  one  of  the  fastest  growing  and  most 
powerful.  The  status  of  QC  models  permits  the 
determination  of  numerous  properties  of  molecules 
and  molecular  complexes  involving  hundreds  of 
atoms.  It  is  important  to  note  that  QC  predictions  for 
small-  and  medium-sized  molecules  and  molecular 
complexes  (tens  of  atoms)  approaches  the  same 
accuracy  as  that  achieved  from  high-quality 
experimental  data.  This  suggests  that  computational 
studies  can  be  more  feasible  and  more  cost  efficient 
than  experimental  investigations. 

QC  Simulations  of  the 
Decomposition  of  Nitrobenzene 

A  team  of  AHPCRC/Jackson  State  University 
researchers  led  by  Dr.  Jerzy  Leszczynski  performed  a 
computer  simulation  of  the  decomposition  of  the 
nitroaromatic  compound,  nitrobenzene,  through 
interaction  with  the  surface  of  an  iron  oxide  (FeO) 
mineral  typically  found  in  soil.  This  process  included 
the  formation  of  the  adsorbed  state,  the 
decomposition  of  nitrobenzene  to  nitrosobenzene, 
and  the  formation  of  the  adsorbed  complex  between 
the  nitrosobenzene  and  the  surface  of  FeO  mineral. 
The  simulation  has  been  performed  using  the  density 
functional  theory  level  of  theory.  The  parallel  version 
of  the  Gaussian-98  package,  updated  for  the  Cray 
T3E,  was  employed.  The  computer  time  necessary 
for  completion  of  this  project  was  an  estimated 
50,000  node  hours  using  up  to  64  nodes  per  job  on 
the  Cray  T3E. 


It  was  concluded  that  the  adsorbed  molecule 
undergoes  destruction  by  transferring  one  of  its 
oxygen  atoms  to  a  surface  iron  atom  in  FeO.  This 
transfer  is  possible  because  the  surface  iron  atoms 
are  not  fully  coordinated.  The  probability  of  such  a 
reaction  pathway  was  determined  by  calculating  a 
value  of  the  activation  energy  for  the  separation  of  an 
oxygen  atom  from  nitrobenzene  and  subsequent 
formation  of  a  new  FeO 
chemical  bond.  It  was 
predicted  that  the  transition 
of  an  oxygen  atom  to  the 
surface  of  iron  oxide  is 
accompanied  by  the 
rotation  of  the  aromatic  ring 
to  form  the  most 
energetically  favorable 
orientation.  This  orientation 
stabilizes  the  structure  by 
means  of  the  formation  of  a 
new  nitrogen-iron  (N...Fe) 
bond.  The  calculated 
activation  energy  is  22.4 
kcal/mol.  The  relatively 
small  size  of  the  activation 
barrier  suggests  that  the  proposed  mechanism  for 
the  decomposition  of  nitrobenzene  is  realistic. 

The  final  step  of  this  project  involved  simulation  of  the 
product  of  nitrobenzene  decomposition: 
nitrosobenzene  adsorbed  onto  iron  oxide.  The 
adsorbed  nitrosobenzene  interacts  with  the  iron 
oxide  surface  through  its  oxygen  and  nitrogen  atoms 
forming  an  adsorption  complex  with  an  adsorption 
energy  of  12.5  kcal/mol.  This  is  almost  one  half  of 
that  calculated  for  the  adsorption  of  nitrobenzene. 
This  result  suggests  that  nitrosobenzene  will  be 
desorbed  from  the  surface  of  iron  oxide  easier  than 
nitrobenzene  after  the  chemical  reaction  of 


“It  is  important  to  note  that 
QC  predictions  for  small- 
and  medium-sized 
molecules  and  molecular 
complexes  (tens  of  atoms) 
approaches  the  same 
accuracy  as  that  achieved 
from  high  quality 
experimental  data.  This 
suggests  that 
computational  studies  can 
be  more  feasible  and  more 
cost  efficient  than 
experimental 
investigations.” 
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Figure  1 .  Reduction  of  nitrobenzene  by  (001)  face  of  iron  oxide 

Applying  QC  methods  allows  computational  chemists 
to  successfully  investigate  not  only  simple  molecules 
but  also  to  extend  their  studies  to  problems  of 
industrial  and  environmental  importance.  The  power 
of  QC  investigations  is  increased  dramatically  when 
modern  parallel  supercomputers  are  used.  In 
particular,  these  simulations  allow  iron  oxide 
containing  minerals  to  be  considered  as  possible 
catalysts  for  the  degradation  of  nitroaromatic 
compounds. 


CTA:  Envionmental  Quality  Modeling  and  Simulation 
Computer  Resources:  Cray  T3E,  IBM  SP2,  and  Cray 
SV1  [AHPCRC] 


decomposition  has  occurred.  The  total  scheme  of 
the  described  transformation  is  presented  in 
Figure  1 . 
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Promoting  Environmental  Stewardship: 
Understanding  Fingering  Flow  in  Unsaturated 
Porous  Media 

J.L.  Nieber  and  A.Y.  Sheshukov 
University  of  Minnesota,  St.  Paul  MN 


n  1998,  the  DoD  cleanup  budget  already 
exceeded  one  billion  dollars1.  While  promoting 
good  stewardship,  this  also  diverts  scarce 
resources  from  other  Defense  priorities. 

Research  that  lowers  cleanup  costs,  and  anticipates 
potential  hazards,  will  have  a  long  ranging  impact  on 
both  the  Department  and  the  welfare  of  US  citizens. 

Many  US  military  sites  have  been  contaminated  by 
fuels,  cleaning  fluids,  and 
munitions.  Researchers 
working  for  the  ARL  are 
seeking  to  understand  the 
movement  of  fluids  through  the 
soil  and  the  deep  unsaturated 
zone;  this  is  important  to  the 
remediation  of  contaminated 
sites,  and  to  predict  the 
behavior  of  fluids  in  any  new 
contamination  events. 

Modeling  fluid  movement  in 
soils  and  deep  unsaturated  zones  involves  solving 
coupled,  nonlinear  mass  balance  equations  that 
quantify  the  movement  of  individual  fluids  in  multi¬ 
phase  fluid  conditions. 

Solving  the  Multi-Phase  Fluid 
Problem 

Conventional  approaches  to  solving  the  multi-phase 
fluid  problem  have  used  mass  balance  equations 
and  assumed  equilibrium  capillary  pressure- 
saturation  relationships.  The  typical  mass  balance 
equation  is  represented  by 

as  a 

— -  V  ■  K{S)Vp  +— K{S)  =  0 
dr  dz 


where  p  is  the  fluid  pressure,  K(S)  is  the  hydraulic 
conductivity,  and  S  is  the  fluid  saturation.  In  the 
conventional  method,  the  fluid  pressure  is  related 
directly  to  the  equilibrium  capillary  pressure- 
saturation  relation  by 

P=P(S) 

where  P  is  the  equilibrium  capillary  pressure. 

In  the  new  approach,  the  ARL  team  of  research 
scientists  kept  the  mass  balance  equation  but 
replaced  the  equilibrium  relation  with  the  non¬ 
equilibrium  relation 


where  x  is  a  relaxation  parameter. 

The  solution  of  the  unconventional  equations  leads  to 
a  completely  new  realm  of  flow  characteristics.  The 
conventional  equations  yield  solutions  that  are 
unconditionally  stable  (in  the  physical  sense).  The 
new  equations  lead  to  flows  that  can  become 
unstable  for  a  wide  range  of  conditions.  Researchers 
concluded  that  the  conditions  leading  to  stability  are 
a  special  case  of  the  more  general  case  where  non¬ 
equilibrium  conditions  prevail. 

Unstable  flow  of  fluids  in  soils  and  the  deep 
unsaturated  zone  is  important  because  the  unstable 
flows  lead  to  inefficient  remediation  of  contaminated 
sites  and  faster  movement  of  contaminants. 

High  performance  computing  resources  are 
important  to  the  study  of  fluid  flow  processes  such  as 
unstable  flows  in  porous  media.  For  two-dimensional 
domains,  it  is  sufficient  to  use  scalar  processor 
computations,  but  for  three-dimensional  domains,  it 


“Unstable  flow  of  fluids 
in  soils  and  the  deep 
unsaturated  zone  is 
important  because  the 
unstable  flows  lead  to 
inefficient  remediation 
of  contaminated  sites 
and  faster  movement 
of  contaminants.” 


1  Guide  to  the  DoD  Environmental  Security  Budget,  October  1998,  DUSD(I&E) 
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Figure  1 .  Distribution  of  fluid  saturation  in  a  two-phase  situation 
showing  the  invasion  of  a  wetting  fluid  into  a  porous  medium 
originally  saturated  with  non-wetting  fluid.  The  invading  fluid  moves 
in  a  fingered  flow  pattern,  representing  a  condition  of  unstable  flow. 
The  fluid  flow  solution  was  derived  using  a  conventional  mass 
balance  equation  for  two-dimensional  flow  with  a  non-equilibrium 
model  for  the  capillary  pressure  -  saturation  relationship. 
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is  necessary  to  perform  computations  on  parallel 
computer  platforms.  At  present,  the  ARLteam  has 
investigated  the  physical  process  for  only  the  two 
dimensional  case,  and  for  homogeneous  porous 
media.  In  the  future,  the  ARL  research  team  will 
continue  to  use  HPCMP  resources  to  consider 
heterogeneous  media,  and  three-dimensional 
domains. 
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Modeling  Structural  Response  to  Blasts 

Joseph  Baum 

Defense  Threat  Reduction  Agency  (DTRA)/SAIC,  Alexandria,  VA 


The  White  House,  the  Capitol,  and  the 
Pentagon  are  three  of  the  most 
prestigious,  recognizable,  and  powerful 
symbols  in  the  United  States.  After  the 
events  of  September  1 1 , 2001 ,  these  buildings  have 
also  become  prime  targets  for  a  terrorist  attack  in  the 
form  of  a  car  or  a  truck  bomb.  Structural  response  of 
these  structures  to  large-size  charges  is  currently 
being  simulated  on  high  performance  computers. 

Dr.  Joseph  Baum  and  a  group  of  research 
associates,  sponsored  by  the  Defense  Threat 
Reduction  Agency  (DTRA),  are  using  high 
performance  computers  and  a  coupled  CFD  and 
Computational  Structural  Dynamics  (CSD)  code  to 
model  all  the  relevant  physical  processes  controlling 
these  scenarios.  These  processes  include:  1) 
detonation  wave  initiation  and  propagation  within  the 
explosive;  2)  CSD  modeling  of  the  car/truck/case 
fragmentation;  3)  blast  wave  expansion  through  the 
breaking  case,  diffracting  about  the  flying  fragments; 
4)  blast  and  fragment  propagation  and  impact  on  the 
target;  5)  target  structural  response  to  the  blast  and 
fragment  loading;  and  6)  the  resulting  new  blast  and 
fragment  environment. 

Using  High  Performance  Computers 
to  Accurately  Model  Fluid-Structure 
Interactions 

This  HPCMP  DoD  Challenge  Project  is  addressing  an 
inability  to  accurately  model  fluid-structure 
interactions  that  require  simultaneous  modeling  of 
such  events  as  a  blast  near  a  building  and  the 
resulting  structural  response,  or  blast  near  a  ship, 
and  the  resulting  holing  and  blast  penetration. 
Researchers  gain  a  better  understanding  of  fluid- 
structure  interactions  by  coupling  CFD  and  CSD 
codes.  For  a  typical  blast  problem,  the  CFD  portion 
models  the  detonation  process  within  the  explosive 
using  the  appropriate  equation-of-state  for  the 
specific  explosive,  and  at  the  same  time,  models  the 
air  blast  that  has  started  propagating.  If  this  is  a 


cased  explosive  (i.e. ,  an  explosive  in  some  steel 
casing — such  as  a  pipe),  the  CSD  part  will  model  the 
breakup  of  the  case.  The  CFD  will  then  model  the 
propagation  in  the  air  of  both  the  expanding  shock 
wave  and  the  fragments  until  they  impact  an  object  (a 
wall  of  a  structure,  a  car,  etc.).  At  that  point,  the  CSD 
code  will  model  the  response  of  the  impacted 
structure  to  this  air  blast  and  fragment  loading.  If,  for 
instance,  the  CSD  model  is  a  ship,  the  CSD  code  will 
model  the  hull  holing  and  fragmentation  and  then  the 
air  blast  going  through  the  breaking/fragmenting 
structure  into  the  next  compartment. 

The  "Glued-Mesh” 

Approach 

The  initial  coupled  CFD/CSD 
algorithm  used  the  so-called 
“glued-mesh”  approach,  where 
the  CFD  and  CSD  faces  match 
identically.  Failure  of  this 
approach  to  model  severe 
structural  deformation,  as  well 
as  crack  propagation  in  steel 
and  concrete,  led  the  team  over 
the  last  year  to  develop  the  so- 
called  “embedded-mesh” 
approach.  Here,  the  CSD  objects  float  through  the 
CFD  domain.  While  each  approach  has  its  own 
advantages,  limitations,  and  deficiencies,  the 
embedded  approach  was  proven  to  be  superior  for 
the  class  of  problems  modeled  here.  The  embedded 
approach  enables  easier  modeling  of  the  response  of 
civilian  structures,  which  include  a  variety  of 
structural  materials,  such  as  brick,  stone,  steel, 
aluminum,  concrete,  and  glass. 

Comparison  to  the  Embedded 
Approach 

Figure  1  shows  a  comparison  between  the  glued 
approach  (Figure  la)  and  the  embedded  approach 
(Figures  1b  to  1e).  The  first  approach  did  notallow 


“The  embedded 
approach  enables 
easier  modeling  of  the 
response  of  civilian 
structures,  which 
include  a  variety  of 
structural  materials, 
such  as  brick,  stone, 
steel,  aluminum, 
concrete,  and  glass.” 
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for  the  application  of  a  physically  correct  crack 
propagation  model,  as  each  crack  had  to  be 
modeled  with  at  least  4-6  fluid  elements  across  the 
narrow  gap,  at  a  prohibitive  computational  cost.  The 
fragments  had  to  be  pre-defined,  hence  precluding 
the  modeling  of  weapons  for  which  there  are  no 
arena  data.  In  addition,  the  break-up  of  each 
fragment  was  a  topology-changing  event,  requiring 
global  remeshing  at  a  fairly  high  computational  cost, 
even  when  performed  in  parallel  using  a  parallelized 
grid  generation  scheme.  In  comparison,  the 
embedded  approach  allows  both  the  treatment  of 
crack  propagation  and  the  formation  of  small 
fragments.  Thus,  the  cracking  model  automatically 
determines  the  size  of  the  fragments.  The  results 
shown  in  Figures  Ic-e  demonstrate:  1)  the 
randomness  of  the  size  of  the  fragments;  2)  the 
narrow  cracks  between  the  fragments,  some  of  which 
are  very  small;  3)  The  large  number  of  fragments 
modeled,  a  simulation  that  would  have  been 
computationally  prohibitive  with  the  glued  approach; 
4)  mesh  adaptation  to  both  the  structure  (smaller 
elements  within  the  cracks  will  yield  more  accurate 
pressure  relief  and  detonation  products  expansion) 
and  density  gradients;  and  5)  most  fragments  reach 
about  the  same  velocity  (within  a  range  of  about 
15%),  in  agreement  with  available  experimental  data. 
After  being  applied  to  the  study  of  blast  and  fragment 
interaction  with  reinforced  concrete  (military) 
structures,  the  new  methodology  is  applied  to  the 
survivability  study  of  important  US  Government 
centers  as  well  as  major  transportation  links  and 
infrastructure  centers. 

A  second  simulation  modeled  frag ment/airb last 
interaction  with  a  steel  wall  chamber.  Similar  to  the 
weapon  case  breakup  case,  three  levels  of  mesh 
refinement  were  used  in  the  CSD  model.  The 
standard  DYNA3D  element  erosion  model  was  used 
to  eliminate  failed  CSD  elements. 

The  numerical  predictions  show  that  the  impacting 
weapon  fragments  arrive  ahead  of  the  airblast, 
punching  holes  through  the  plate  (Figure  2a).  Next, 
the  pressure  blast  from  the  detonation  tears  the 
weakened  plate  apart  (Figure  2b).  The  eroded  plate 
elements  were  converted  into  particles  that  can 
interact  with  the  rest  of  the  structure.  Contact 
conditions  were  enforced  between  all  entities  to 
avoid  fragment  interpenetrations  and  therefore  a 
failure  in  meshing. 

The  results  will  give  researchers  a  better 
understanding  of  the  structural  response  of  a  variety 
of  materials  to  a  car  or  truck  bomb.  This  will  enable 
engineers  to  design  buildings  which  are  more 
resistant  to  these  types  of  blasts. 


(e) 


Figure  1 .  Comparison  of  glued  and  embedded  CSD  schemes 
applied  to  the  weapon  fragmentation  study 
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Figure  2.  Airblast/fragments  impact  on  a  steel  plate 


-25- 


-  Protect  Bases  of  Operation 


Evaluation  of  Blast  Response  of  the  Pentagon 

Tommy  Bevins,  Byron  Armstrong,  James  Baylot,  and  James  O’Daniel 

US  Army  Corps  of  Engineers,  Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 


Protecting  our  military  personnel  from 

terrorist  attacks  is  critical  to  DoD  mission 
success.  Real  life  scenarios  rarely 
consist  of  simply  an  explosion,  the 
propagation  of  the  blast  load  over  a  flat  terrain,  and 
its  impact  on  a  structure.  The  blast  environment  is 
affected  by  the  presence  of  other  structures,  as  well 
as  the  natural  topography.  A  complete 
understanding  of  this  complex  blast  environment, 
including  the  effects  of  adjacent  structures  on  the 
blast  loads  on  a  structure,  is  needed  to  protect 
building  occupants  in  an  urban  environment.  These 
effects  were  studied  in  a  combined  experimental/ 
analytical  study  conducted  at  the  Engineer  Research 
and  Development  Center  (ERDC).  The  analytical 
study  used  high-priority  HPC  processing  hours 
provided  under  the  HPCMPO  DoD  Challenge  Project: 
“Evaluation  and  Retrofit  for  Blast  Protection  in  Urban 
Terrain.”  Simulations  of  several  different  experiment 
geometries  and  combinations  of  structures  have 
been  performed  as  part  of  this  DoD  Challenge 
Project.  The  analyses  have  compared  very  well  with 
experimental  results,  giving  credibility  to  these 
analysis  methods  for  use  in  further  blast  propagation 
studies. 

The  Defense  Department  placed  a  high  priority  on 
the  rapid  repair  of  the  damages  to  the  Pentagon 
resulting  from  the  September  1 1 , 2001 ,  terrorist 
attack.  The  US  Army  Corps  of  Engineers  had  been 
tasked  with  providing  possible  retrofit  designs  to  the 
Pentagon  in  order  to  improve  its  response  to  a  range 
of  terrorist  threats.  Because  of  the  desire  to  rapidly 
repair  the  Pentagon,  the  Corps  of  Engineers 
recommendations  for  improvements  had  to  be 
finalized  and  presented  to  the  Pentagon  Renovation 
Office  (PenRen)  within  eight  weeks.  This  effort 
involved  determining  the  loads  on  the  components  of 
the  Pentagon  and  the  response  of  these  components 
to  the  loads.  It  also  involved  developing,  analyzing, 
and  evaluating  retrofit  concepts  to  improve  safety  of 
building  occupants. 


Prediction  of  Loads  and  Response  of 
Pentagon 

The  Pentagon  is  a  complex  structure  and  in  some 
respects  behaves  like  a  group  of  structures  rather 
than  one.  There  are  five  concentric  pentagonal  rings 
separated  by  a  lightwell  (air  space)  over  the  top  three 
floors  of  adjacent  rings.  The 
lightwell  walls  contain 
hundreds  of  windows.  If  a 
terrorist  bomb  is  detonated 
outside  the  Pentagon,  the 
blast  pressures  will  propagate 
over  the  top  of  the  Pentagon 
and  into  the  lightwells,  thus 
loading  the  windows.  The 
propagation  of  blast  into  the 
lightwells  is  similar  to  the 
propagation  of  blast  over, 
around,  and  between 
structures.  An  accurate 
determination  of  the  loads  on 
these  lightwell  windows  will 
allow  the  Department  to 
improve  the  structure  and  consequently,  protect  its 
occupants.  Figure  1  shows  that  airblast  propagation 
into  the  lightwells  could  be  a  hazard  to  occupants  of 
rooms  near  the  lightwells. 

The  Pentagon  is  a  concrete  frame  structure  with 
nonstructural  exterior  in-fill  walls.  The  in-fill  walls  of 
the  structure  are  two-layer  brick  walls  clad  with  large 
limestone  blocks.  Many  large  windows  are  present 
on  the  first  four  floors  of  the  outer  wall  of  the 
Pentagon.  Multiple  scenarios  were  evaluated, 
including  the  existing  Pentagon  exterior  walls,  the 
current  Pentagon  retrofit  walls,  and  the  enhanced 
retrofit  designs. 

Research  conducted  under  the  challenge  project  has 
demonstrated  that  the  HPC  simulations  match  the 
experiments  well.  Further  simulations  will  be 
performed  beyond  those  for  which  experiments  will 


“This  research  is 
important  to  DoD 
because  it  transitions 
technology  needed  to 
improve  the  survivability 
of  the  Pentagon  to  those 
responsible  for  its  blast 
retrofit.  The  research 
also  decreases  costs  by 
helping  to  determine  the 
critical  areas  of  the 
Pentagon  so  that 
resources  are  expended 
in  those  locations  where 
they  are  most  needed.” 
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Figure  1 .  Blast  propagation  into  lightwells 


be  conducted.  The  results  will  be  used  to  develop 
engineering  models  for  blast  in  urban  terrain. 

HPC  simulations  were  used  to  evaluate  the  safety  of 
the  Pentagon  occupants  to  a  range  of  terrorist 
threats.  The  retrofits  currently  installed  in  portions  of 
the  Pentagon  as  well  as  other  improved  retrofits  were 
evaluated.  Figure  2  shows  that  the  bricks  of  the 
exterior  wall  will  be  a  hazard  to  occupants  for  the 
threat  analyzed.  The  brick  walls  of  the  retrofitted 
Pentagon  will  fail  as  shown  in  Figure  3,  but  will  be 
captured  by  the  geotextile  fabric  and  will  not  be  a 
hazard.  The  results  of  the  study  were  presented  to 
PenRen,  and  PenRen  has  asked  the  ERDC  to 
participate  in  a  study  to  enhance  the  safety  of 
Pentagon  occupants. 

This  research  is  important  to  DoD  because  it 
transitions  technology  needed  to  improve  the 
survivability  of  the  Pentagon  to  those  responsible  for 
its  blast  retrofit.  The  research  also  decreases  costs 
by  helping  to  determine  the  critical  areas  of  the 
Pentagon  so  that  resources  are  expended  in  those 
locations  where  they  are  most  needed.  This  research 


Figure  2.  High-hazard 
failure  of  original 
Pentagon  wall 


Figure  3.  Response  of 
successfully  retrofitted 
Pentagon  wall 


promotes  the  understanding  of  experiments  and  the 
phenomenology  of  the  blast  environment,  which  will 
in  turn  lead  to  better  engineering  models  for 
predicting  blast  loads. 

Engineering  Models  Too  Simple 

Blast  loadings  on  structures  have  typically  been 
evaluated  using  empirical  equations  that  relate  blast 
pressure  characteristics  (incident  pressure,  incident 
impulse,  reflected  pressure,  reflected  impulse,  shock 
front  velocity,  time  of  arrival,  and  dynamic  pressure) 
to  known  physical  parameters  (explosive  charge 
weight,  standoff,  and  the  angle  between  the  shock 
front  propagation  direction  and  the  normal  to  the 
structure).  These  relationships  have  been 
determined  for  both  hemi-spherical  surface  bursts 
and  for  spherical  bursts  in  free  air.  These  equations 
assume  that  there  are  no  obstacles  between  the 
explosive  source  and  the  target.  The  simplest 
approach  to  determine  the  loads  on  a  structure 
behind  an  obstacle  is  to  assume  that  the  obstacle 
has  no  effect  on  the  pressure  on  the  structure  behind 
it.  A  second  approach  adjusts  the  standoff  distance 
to  account  for  the  distance  that  must  be  traveled  to 
go  around  the  obstacle.  Neither  of  these  methods 
provides  reliable  or  precise  estimates  of  the 
pressures  time-history  on  a  shielded  target.  One 
problem  is  that  the  relationship  between  the  incident 
pressure,  the  shock  front  velocity,  and  the  reflected 
pressure  is  not  the  same  for  a  clear  path  to  a  target 
as  it  is  for  an  obstructed  path  to  a  target.  Recent 
experiments  have  demonstrated  that  even  if  the 
incident  pressure  at  a  target  point  is  essentially  the 
same  with  and  without  an  obstruction,  the  shock  front 
velocity  at  the  target  may  be  significantly  different, 
leading  to  a  significantly  different  reflected  pressure. 

The  response  of  masonry  walls  to  blast  is  typically 
modeled  using  single-degree-of-freedom  (SDOF) 
representations  of  the  wall  to  determine  a  pressure- 
impulse  (P-l)  diagram.  The  applied  pressure  and 
impulse  can  then  be  plotted  on  the  diagram.  If  that 
pair  is  above  and  to  the  right  of  the  curve,  then  the 
response  exceeds  the  damage  level  associated  with 
the  P-l  curve.  This  type  of  analysis  is  valid  and  easily 
applicable  to  simple  masonry  walls  such  as  a  single 
layer  brick  wall.  For  complicated  structures  where  a 
brick  wall  interacts  with  another  wall,  these  methods 
can  only  be  used  as  a  guideline,  unless  validated  by 
some  other  means. 
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Explicit  Time-Stepping  Method 
Models  Blast  Response 

With  the  availability  of  HPC,  it  is  possible  to  calculate 
the  problem  of  shock  propagating  through  the  air  by 
dividing  the  large  volume  —  including  the  structure, 
the  air,  and  the  explosives  —  into  a  large  number  of 
small  cells.  The  propagation  of  shock  is  modeled  in 
an  explicit  time-stepping  method.  The  method 
conserves  mass  and  momentum  and  represents  the 
correct  equations  of  state  for  the  air  and  the 
explosives.  Determining  the  solution  requires  the 
computation  of  millions  of  equations  for  each  of 
thousands  of  time  steps.  HPC  resources  allow  these 
calculations  to  be  performed  in  a  finite  length  of  real¬ 
time  for  significant  amounts  of  simulation  time.  The 
advantage  of  this  method  over  the  usual  method  for 
obtaining  loads  is  that  the  incident  pressure,  shock 
front  velocity,  dynamic  pressure,  and  reflected 
pressure  are  all  computed  based  on  the  physics  of 
the  problem  rather  than  on  the  combination  of 
empirical  equations  that  are  not  actually  valid  for  the 
problem  being  solved.  Historically,  howto 
computationally  solve  the  governing  equations  was 
known,  but  there  was  not  enough  computer  power  to 
solve  significant  problems  in  a  practical  length  of 
actual  time. 

Simplified  methods  provided  an  overall  answer  to  the 
response  of  the  structure,  whereas  these  methods 
provide  a  solution  with  detailed  information  about  the 
response.  HPC  makes  it  possible  to  explicitly  model 
the  interaction  of  individual  bricks  of  a  brick  wall. 
Using  procedures  that  have  been  developed  at 
ERDC,  the  model  of  the  complex  wall  system  of  the 
Pentagon  was  developed.  The  model  included  the 
two-layer  brick  wall,  the  concrete  frame,  and  the 
limestone  fagade.  Models  were  also  developed  for 
several  retrofit  designs.  Approximate  P-l  diagrams  for 
several  wall  configurations  were  validated  using  HPC 
simulations.  Scientific  visualization  has  been  used 
extensively  to  assist  in  understanding  the 
phenomenology  of  blast  interacting  with  structures 
and  the  response  of  those  structures  to  blast. 


some  of  the  deficiencies  identified  by  the  HPC 
analyses  performed  in  the  original  study. 

Experimental  results  are  very 
useful  for  studying  blast  loads 
on  structures,  but  are  not 
sufficient  to  completely 
understand  the 
phenomenology  of  blast 
passing  around  an  obstacle 
and  then  loading  a  structure. 

HPC  results  were  used  to  fill 
in  gaps  in  the  data  and  to 
visualize  how  the  physical 
process  evolves. 

Continued  expansion  of  HPC  resources,  including 
number  of  processors  available  and  the  compute 
speed  of  those  processors,  allowed  the  size  of  the 
models  to  expand.  By  allowing  the  use  of  a  larger 
number  of  cells  or  elements,  more  detail  could  be 
inserted  into  the  models  and  fewer  simplifications 
needed  to  be  taken.  All  of  these  models  were 
approximations,  and  the  capability  to  more  explicitly 
model  the  subject  allowed  more  accurate  solutions  to 
be  obtained. 
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“Historically,  how  to 
computationally  solve  the 
governing  equations  was 
known,  but  there  was  not 
enough  computer  power 
to  solve  significant 
problems  in  a  practical 
length  of  actual  time.” 


Initial  Blast  Load  Evaluations 
Completed 

The  availability  of  high-priority  HPC  time  has  allowed 
ERDC  to  complete  initial  evaluations  of  the  methods 
being  used  to  study  blast  loads  on  structures  in 
urban  terrain.  These  initial  evaluations  have  been 
done  in  time  to  impact  the  PenRen  Program.  The 
evaluations  are  also  very  encouraging  and  give 
credibility  to  the  analyses  to  determine  the 
survivability  of  the  Pentagon  to  a  terrorist  attack.  The 
PenRen  asked  the  ERDC  to  assist  in  overcoming 
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Modeling  Methods  for  Assessment  of  Gun 
Propellant  Vulnerability 

John  Starkenberg  and  Toni  Dorsey 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 


In  order  to  increase  the  muzzle  energy  in 
medium-  and  large-caliber  guns  (and 
consequently  their  lethality),  approaches 
involving  increasing  the  available  mass  of  gun 
propellant  are  being  pursued  by  scientists  at  the 
ARL.  This  generally  means  packing  more  propellant 
into  the  available  volume,  which  may  adversely  affect 
the  vulnerability  of  associated  weapons  systems  by 

allowing  the  propellant  bed 
to  support  detonation. 

Starkenberg  and  Dorsey  are 
developing  modeling 
approaches  for  assessing 
the  violence  of  the  reactive 
responses  of  propellant  beds 
to  unplanned  stimuli, 
emphasizing  the  tendency 
for  detonation  to  spread  from 
one  part  of  a  bed  to  another. 
Their  results  will  be  applied 
to  emerging  propellant 
technologies  to  determine  how  to  design  beds  to 
minimize  reactive  response  violence. 

Studying  Detonation  Propagation 
Models 

Traditionally,  gun-propellant  vulnerability  has  been 
assessed  through  experiments,  not  simulations.  Now 
explosive  initiation  models  such  as  those  included  in 
the  Eulerian  finite-volume  continuum  mechanics 
solver  CTH  may  also  be  used  to  model  detonation 
propagation  and  failure.  A  detailed  computational 
study  of  the  use  of  these  models  for  detonation 
propagation  was  undertaken.  Based  on  the 
understanding  gained  in  that  study,  large 
three-dimensional  simulations  of  the  spread  of 
detonation  between  propellant  grains  and  between 
parts  of  more  advanced  charge  designs  were 


undertaken  using  CTH’s  History  Variable  Reactive 
Burn  (HVRB)  model. 

The  fundamental  study  showed  that  extremely  fine 
computational  resolution  and  special  calibrations  of 
the  HVRB  model  are  required  to  accurately 
determine  the  smallest  propellant  grain  diameter  that 
allows  a  detonation  to  propagate.  This  is  referred  to 
as  the  failure  or  critical  diameter.  Eight  different 
propagation  and  contact  modes  for  propagation 
between  pairs  of  cylindrical  propellant  grains  in  a  bed 
were  identified  and  simulated.  Figure  1  shows  the 
results  of  a  simulation  of  grains  in  side-to-side  point 
contact  with  the  detonation  spreading  directly  from 
one  grain  to  the  other.  The  results  show  that 
diameters  larger  than  the  critical  diameter  are 
required  for  grain-to-grain  propagation  and  explain 
experimental  observations.  This  indicates  that 


Figure  1 .  Sequence  of  plots  showing  propagation  of  detonation 
between  propellant  grains 


“The  modeling 
approaches  developed  in 
this  program  are  being 
used  for  the  Army’s  Future 
Combat  System  to  help 
design  high-loading- 
density  gun-propelling 
charges  to  provide 
maximum  performance 
with  minimum  impact  on 
systems  survivability.’’ 
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detonation  can  be  prevented  from  spreading 
throughout  a  bed  even  when  the  grains  are  larger 
than  the  critical  diameter.  Results  from  simulations  of 
more  advanced  propellant  bed  designs  also  show 
how  the  beds  can  be  designed  to  minimize  the 
violence  of  the  reactive  response. 

Modeling  Used  to  Design  Propellant 
Charges 

The  modeling  approaches  developed  in  this  program 
are  being  used  for  the  Army’s  Future  Combat  System 
to  help  design  high-loading-density  gun-propellant 
charges  to  provide  maximum  performance  with 
minimum  impact  on  systems  survivability.  The 
approach  is  suitable  for  evaluating  special  propellant 
configurations  even  when  samples  for  testing  are 
unavailable.  This  helps  in  understanding  the  system 
vulnerability  implications  of  using  specific  propellant 
configurations  on  proposed  weapons  platforms. 
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Robust  Scalable  Information  Processing  for 
Multistatic  Sonars 

Jeffery  Dugan,  Carol  Nelepovitz,  and  Daniel  Sternlicht 
Office  of  Naval  Research  (ONR),  Arlington,  VA 


Acoustic  sonar  performance  for  various 

Anti-Submarine  Warfare  programs  benefits 
from  improved  algorithms  that  increase 
detection  of  the  submarine  threat,  as  well  as 
reduce  the  false  alarms  caused  by  misclassified 
clutter  detections.  Current  algorithm  development 
achieves  robustness  by  analyzing  data  from  a  large 
number  of  at-sea  tests,  which  represent  different 
environments,  target  dynamics,  and  transmitted 
signals  (Figure  1). 


Figure  1 . 

Surveillance 

Platforms 


Active  sonar  systems  benefit  from  the  ability  to 
estimate  target  range.  Complex  pings  can  be  used 
to  improve  the  target  state  estimation  by  providing  a 
range-rate  estimate.  One  algorithm  tested  and 
validated  using  the  HPC  resources  is  an  improved 
range  rate  estimator  for  one  of  the  mainstay 
waveforms  in  active  sonar,  the  Hyperbolic  Frequency 
Modulated  (HFM)  waveform.  Improved  range  rate 
estimation  reduces  the  percentage  of  high-speed 
clutter  thus  reducing  both  the  algorithmic  and 
operator  resources. 


Correlation-Based  Algorithms 

Traditionally,  range  rate  estimates  for  HFM  waveforms 
have  been  calculated  using  a  correlation  based 
algorithm.  The  range  rate  estimates  from  the 
correlation-based  algorithm  tended  to  incorrectly 
estimate  the  range  rate  of  bottom-induced  clutter.  In 
the  system,  there  are  multiple  channels.  The 
correlation-based  algorithm  uses  pair  wise 
correlation  of  the  HFM  echo  returns  to  update  the 
time  delay  of  the  HFMs.  The  correlation  information 
is  reduced  to  two  time  delays.  The  two  time  delays 
are  then  used  in  a  line  fit,  and  the  slope  of  the  line  is 
used  to  estimate  the  range  rate.  The  existing  method 
loses  the  shape  information  of  the  echoed  HFM,  and 
spurious  noise  spikes  can  cause  the  line  fit  to  fit  at  an 
inappropriate  speed  estimate. 

Improved  Algorithm  Analysis 

The  HPC  resources  are  used  to  improve  algorithm 
analysis,  with  a  goal  of  transitioning  the  algorithms 
into  fleet  use.  The  Space  and  Naval  Warfare 
(SPAWAR)  System  Center  (SSC)  Signal  Processor 
and  the  ORINCON  Information  Processor  (IP),  hosted 
on  the  HPC  HP-V2500  Multiprocessor  Computer,  are 
flexible  and  can  be  configured  for  processing  data 
from  various  acoustic  arrays  (Figure  2). 

The  HPC  resources  enable  efficient  and  robust 
algorithm  evaluation  due  to  the  amount  of  data  that 
can  be  stored  and  processed.  The  Scalable 


Signal  and  information  processing  streams 

Figure  2.  Signal  and  Information  Processing  Streams 
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Programming  Environment  (SPE)  available  on  the 
V2500  provides  a  well-defined  interface,  portability, 
and  reduced  development  time.  The  SPE  layer 
allows  for  modularization,  and  adding  a  module  to 
the  system  is  relatively  easy.  Algorithms  that  are 

being  compared  may  be  run  in 
parallel,  and  their  classifier 
Receiver  Operating 
Characteristics  (ROC)  curves 
can  be  analyzed  quickly. 

HPC  allowed  this  research 
project  to  robustly  test  and 
evaluate  the  improvement  in 
active  sonar  detection  and 
classification  when  applying 
ORINCON’s  improved  range 
rate  algorithm. 

The  SPAWAR  System  Center, 
San  Diego  (SSCSD)  Advanced 
Visualization  for  Intelligence, 
Surveillance,  and  Reconnaisance  (ADVISR) 
visualization  tool  has  been  used  in  some  of  the  work 
as  well  as  in  MATLAB.  MATLAB  has  been  an 
invaluable  tool  for  data  analysis  and  prototyping.  The 
ADVISR  simulations  allowed  for  viewing  the  evolution 
of  the  situation  overtime  and  provided  improved 
viewer  comprehension. 


classification  calculations  are  performed  and  target 
or  clutter  likeness  is  determined. 

For  the  data  analyzed,  target  detections  for  surface 
and  sub-surface  detections  were  achieved  at  tactical 
ranges.  The  improved  range  rate  parameter  is 
calculated  in  the  detection  portion  of  the  IP.  Analysis 
of  the  improved  range  rate  parameter  has  shown  a 
marked  reduction  in  the  range  rate  of  high-speed 
clutter  detections  while  preserving  the  range  rate  of 
targets  of  interest.  As  shown  in  the  results  plot 
(Figure  4),  the  algorithm  provides  a  minimum  61 
percent  reduction  in  “high-speed”  clutter  detections. 
At  a  lower  range-rate  XI ,  which  is  half  of  X2,  over  50 
percent  of  the  clutter  detections  are  removed.  This 
improvement  reduces  the  single-ping  false  alarm  rate 
when  alerting  the  operator  to  detections  that  are 
thought  to  be  from  moving  objects.  The  importance 
of  this  project  is  the  capability  to  robustly  test 
candidate  algorithms.  The  availability  of  the  HPC 
resources  allowed  use  of  the  SPE.  The  SPE  allowed 
for  the  design  of  the  processing  components  in  a 
modular,  scalable  fashion.  This  architecture  provided 
a  test  bed  for  active  signal  processing  that  allows 
easy  replacing  of  various  candidate  classification 
algorithms.  The  large  storage  and  computing 
capabilities  allowed  very  large  quantities  of  data  to  be 
processed,  allowing  robust  testing. 


"The  HPC  resources 
are  used  to  improve 
algorithm  analysis,  with 
a  goal  of  transition  of 
algorithms  into  fleet 
use.  The  HPC 
resources  enable 
efficient  and  robust 
algorithm  evaluation 
due  to  the  amount  of 
data  that  can  be 
stored  and 
processed.’’ 


Target  Detections  Improved 

Processing  within  the  Scalable  Signal  Processor  and 
Active  Information  Processor  consists  of  several 
steps.  The  echoes  for  the  HFM  transmissions  are 
beam  formed,  matched  filtered,  noise  normalized, 
and  thresholded.  The  IP  receives  the  threshold 
crossings  and  attempts  to  group  together  crossings 
that  are  from  the  same  echo  source.  This  is  the 
clustering  portion  of  the  IP.  The  next  step  is  to 
combine  the  multiple  HFM  waveforms  that  were 
transmitted  into  detections.  Detections  use  an  M-out- 
of-N  criterion  where  M  is  the  minimum  number  of  the 
multiple  HFM  sweeps  and  N  is  the  number  of  HFM 
sweeps  transmitted.  These  HFM  detections  can  be 
combined  with  other  waveform  types,  such  as 
continuous  waveform,  to  form  wave  train  detections 
(Figure  3).  The  wave  train  detections  are  sent  to  the 
tracker  for  linking  over  time.  At  all  stages, 


Freq. 

(Hz.) 


Wavetrain  comprised  of  three  HFM  waveforms 


Percent  Reduction  of  Clutter  above  and  X?  Kts. 


Figure  4.  Range  Rate  Results 


Estimating  Target  Range  Rates 

In  this  environment  and  architecture,  Navy 
researchers  have  tested  a  novel  approach  to 
estimating  target  range  rate  with  HFM  waveforms  that 
are  not  typically  associated  with  providing  range  rate. 
In  addition  to  estimating  the  range  rate  of  a  detection, 
they  also  computed  a  new  classification  clue  “t-ratio” 
which  is  the  speed  estimate  divided  by  the  standard 
deviation  of  the  speed  estimate.  The  t-ratio 
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computation  represents  the  confidence  of  the  range 
rate  estimate.  Adding  t-ratio  to  the  classification  clue 
set  resulted  in  a  very  significant  reduction  in  the 
number  of  clutter  detections  that  would  be  classified 
as  targets  and  sent  to  an  operator,  thus  reducing  their 
workload  and  allowing  them  to  focus  on  other 
detections,  while  preserving  the  range  rates  of 
targets  of  interest. 
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Detached-Eddy  Simulation  of  Massively  Separated  Flows  over  Aircraft  by  Kyle  Squires,  James  Forsythe,  Scott  Morton, 
Kenneth  Wurtzler,  William  Strang,  Robert  Tomaro,  and  Philippe  Spalart 

Flow  over  the  delta  wing  at  21°  angle  of  attack.  Isosurface  of  vorticity 

Airdrop  Systems  Modeling:  Fluid-Structure  Interaction  Applications  by  Richard  Benney,  Keith  Stein,  Tayfun  Tezduyar, 
Michael  Accorsi,  and  Hamid  Johari 

This  figure  shows  the  structural  response  of  the  upper  canopy  as  it  deforms  and  interacts  with  the  surrounding  flow  field.  The 
colors  on  the  canopy  correspond  to  the  differential  pressures,  which  are  a  part  of  the  fluid  dynamics  solution. 

Absolute  Scalability  of  Space/Time  Domain  Decomposition  in  Computational  Structural  Dynamics  for  Virtual  Modeling  and 
Testing  Applications  by  R.  Kanapady  and  K.  K.  Tamma 

Solid  modeling  of  keel  beam 

Accurate  Modeling  of  Lossy  Frequency  Selective  Surfaces  for  Real-Time  Design  by  John  Meloling,  Wendy  Massey,  and 
David  Hurdsman 

Smith  chart  of  boundary  admittance  using  past  approximations  (green)  and  more  accurate  solutions  (blue) 
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“All  warfare  is  based  on  deception.  Hence,  when  able  to  attack,  we  must  seem  unable;  when  using 
our  forces,  we  must  seem  inactive;  when  we  are  near,  we  must  make  the  enemy  believe  we  are  far 
away;  when  far  away,  we  must  make  him  believe  we  are  near.”1  The  following  success  stories  describe 
some  important  efforts  to  use  HPC  for  the  design  and  analysis  of  military  platforms  and  the  battlespace 
environment  to  help  project  and  sustain  US  forces. 

High  performance  computing  is  critical  to  the  research,  design,  acquisition,  and  testing  of  military 
platforms  with  superior  warfighting  capabilities.  One  of  the  primary  goals  of  these  warfighting 
platforms  is  to  project  US  forces  into  hostile  situations  and  sustain  them  in  that  situation. 

High  performance  computing  contributes  significantly  to  the  design  of  these  weapons  platforms  in  a 
variety  of  ways,  as  chronicled  by  the  first  group  of  seventeen  success  stories.  What  is  particularly 
impressive  is  the  breadth  of  computational  work  represented  in  terms  of  length  scales.  They  range 
from  the  nanoscale  as  detailed  in  the  Jarvis  and  Carter  accomplishment  of  using  quantum  mechanics 
to  investigate  thermal  barrier  coatings  on  jet  engine  turbine  blades  to  the  size  of  large  vehicles  as 
discussed  in  the  Gorski,  Hyman,  Taylor,  and  Wilson  article  on  hydrodynamics  on  entire  ships. 
Component  level  simulations  of  armor  and  composites  are  discussed  in  articles  by  Holmquist  and 
Johnson,  and  Shires,  Henz,  and  Mohan.  Analysis  of  components  such  as  turbomachines  and  full 
systems  such  as  modern  airplanes  is  performed  with  Computational  Fluid  Dynamic  (CFD)  simulations 
in  articles  by  Moin,  You,  Wand,  and  Mittal;  by  Jurkovich,  Rittinger,  and  Weyer;  by  Polsky;  by  Rizzetta; 
by  Green  and  Ott;  and  by  Forsythe,  Morton,  Squires,  Wurtzler,  Strang,  Tomaro,  and  Spalart.  Pushing 
the  computational  frontiers  downward  in  size  to  the  nanoscale  to  model  specific  surfaces  atomistically 
or  upward  in  size  to  model  aerodynamic  flow  in  a  time-dependent  fashion  around  whole  aircraft 
represents  an  expansion  of  the  computational  capability  envelope  that  is  now  just  becoming  possible 
with  present-day  systems. 

Along  another  dimension,  this  set  of  success  stories  represents  several  of  the  program’s  major 
computational  areas.  In  addition  to  CFD,  there  are  also  stories  in  computational  chemistry  and 
materials  science,  computational  structural  mechanics,  and  computational  electromagnetics  and 
acoustics.  The  stories  by  Hill,  Navale,  and  Sancer,  and  by  Mark,  Clarke,  Namburu,  Le,  and  Nehrbass 
employ  the  techniques  of  computational  electromagnetics  (CEM)  to  compute  the  electromagnetic 
signatures  of  electromagnetically  large  combat  vehicles.  Of  particular  interest  is  interdisciplinary 
computational  work,  as  presented  in  the  Soo,  Feiz,  and  Menon  contribution  on  incorporation  of  finite 
rate  chemistry  into  computational  fluid  dynamic  studies  of  fuel  injectors  for  propulsion  systems,  and 
the  coupled  fluid/structure  investigations  of  aircraft  wing  response  to  artillery  damage  of  Hinrichsen. 
The  interaction  of  structures  with  the  surrounding  land,  air,  and  sea  environments  is  investigated  in  the 
article  by  Prakash  on  testing  armor,  in  the  article  by  Benney,  Stein,  Tezduyar,  Accorsi,  and  Johari  on 
airdrop  systems,  and  in  the  article  on  the  effects  of  mines  by  Landsberg. 

All  in  all,  these  seventeen  stories  make  impressive  contributions  to  solving  real  DoD  problems.  Every 
single  one  of  them  addresses  a  real  mission  need,  ranging  from  a  specific,  short-term  problem  on  a 
current  weapon  system  through  basic  research  and  design  work  on  future  weapons  systems. 

Synchronization  and  coordination  within  the  battlespace  to  project  and  sustain  US  Forces  are  key 
transformational  military  goals.  The  2001  Quadrennial  Defense  Review  states  that  DoD  will  “...  develop 
an  interoperable,  joint  C4ISR  architecture  and  capability  that  includes  a  tailorable  joint  operational 
battlespace.”  This  battlespace  is  defined  by  “layers”  of  information  where  some  layers  are  static  (e.g., 


1  SunTzu,  The  Art  of  War 
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land,  buildings,  space)  and  others  are  dynamic  (e.g.,  weather,  ocean,  foliage,  weapons  locations).  The 
battlespace  environment  itself  is  the  subject  of  study  the  next  group  of  five  stories  that  discuss 
specifically  how  the  Common  High  Performance  Computing  Software  Support  Initiative  (CHSSI)  and 
DoD  HPC  efforts  help  build  the  dynamic  weather/ocean  layers. 

Characterizing  the  environmental  battlespace  is  the  most  important  goal  in  building  dynamic  weather/ 
ocean  layers.  But  what  does  “characterizing  the  environmental  battlespace”  really  mean?  It  is  the 
ability  to  take  real-time  weather/ocean  observations,  combine  the  observations  with  numerical  model 
fields,  and  create  a  volumetric  dataset  where  weather/ocean  layers  are  defined  in  space  and  time. 
Allard  et.  al.,  provides  an  overview  on  how  their  CHSSI  portfolio  efforts  are  building  the  ocean,  tide, 
wave,  estuarine,  and  riverine  layers  for  littoral  regions.  Through  new  techniques  in  coupling  different 
models  to  create  a  synthesized  environment,  the  littoral  battlespace  layers  will  constantly  provide  the 
warfighters  with  the  appropriate  readiness  to  conduct  their  missions. 

The  Fleet  requires  wave  and  surf  hind-casts  and  short-term  forecasts  for  accurate  mission  planning. 
Successful  amphibious  assaults  are  enabled  by  accurate  wave  modeling  along  the  beach.  Smith 
describes  how  HPC  resources  were  critical  to  building  the  “wave”  layer  within  the  littoral  battlespace 
portfolio.  The  integration  effort  described  by  Allard  et.  al.,  involves  the  coupling  of  the  wave  layer  to 
other  littoral  layers,  and  the  fuzing  of  the  layers  (in  time  and  space)  to  create  a  volumetric  dataset. 
Burnett  focuses  on  the  atmospheric  layer  and  describes  a  successful  story  of  how  HPC  resources 
allowed  the  operational  community  to  continue  providing  operational  weather  layers  to  the  warfighter 
while  the  Production  Center  converted  from  vector  to  scalable  architectures.  The  story  by  Hughes, 
Wegiel,  McAtte,  Michalakes,  Barker  and  LaCroix  discusses  integration  of  data  from  real-time  weather 
observations  with  HPC  forecast  models. 

The  final  critical  part  in  developing  the  environmental  battlespace  is  the  visualization  and 
understanding  of  the  various  layers.  Gruzinskas  describes  how  HPC  efforts  in  visualizing  the 
battlespace  environment  will  ensure  that  planners  and  decision-makers  have  an  in-depth 
understanding  of  the  constantly  changing  environmental  conditions.  He  describes  a  particular  case 
where  the  Navy  supported  the  Ehime  Maru  salvage  operations  using  virtual  environments  and  on¬ 
scene  data.  The  effort  actually  went  a  step  further  and  integrated  real-time  observations  with  the  virtual 
environment. 

Characterizing  the  battlespace  environment  is  an  on-going  effort.  These  success  stories  show  that  the 
Meterology  and  Oceanography  Community  is  only  a  third  of  the  way  toward  reaching  their  goal  of 
creating  a  volumetric  battlespace.  Initial  steps  in  building  the  littoral  environmental  battlespace  and 
crucial  DoD  HPC  support  of  CHSSI,  Programming  Environment  and  Training  (PET),  and  MSRCs  will 
allow  the  community  to  reach  their  goal  within  the  next  five  years. 


Dr.  William  Burnett 

Naval  Meteorology  and  Oceanography  Command,  Stennis  Space  Center,  MS 
Dr.  Larry  P.  Davis 

HPCMP  Office,  Arlington,  VA 
Dr.  Robert  E.  Peterkin,  Jr. 

HPCMP  Office,  Arlington,  VA 
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How  Quantum  Mechanics  Can  Impact  Jet  Engines: 
Improving  Aircraft  Design  at  the  Nanoscale 

Emily  A.  Jarvis  and  Emily  A.  Carter 

University  of  California,  Department  of  Chemistry  and  Biochemistry  Los  Angeles,  CA 
Sponsor:  Air  Force  Office  of  Scientific  Research  (AFOSR) 


Thermal  barrier  coatings  (TBCs)  of  jet 
engine  turbines  afford  thermal  and 
oxidative/corrosive  protection  of  the 
underlying  metal  superalloy  that  is  crucial  to 
aircraft  performance.  This  coating  must  withstand 
temperature  cycling  over  a  range  exceeding  1000  °C, 
as  the  engine  is  taken  from  rest  to  maximum 
operating  capacity.  The  highest  temperatures 
accessed  within  the  combustion  chamber  are  even 
beyond  the  melting  point  of  the  superalloy  substrate. 
Nevertheless,  such  high  temperature  combustion  is 
required  to  attain  optimal  engine  power  and 
maximum  fuel  efficiency. 


TBC  Spallation 

Unfortunately,  a  process  known  as  spallation, 
whereby  the  TBC  peels  off  the  underlying  substrate, 
limits  a  TBC’s  operational  lifetime.  TBC  spallation  is 
catastrophic  because:  1)  the  superalloy  composing 
the  turbine  blade  is  left  unprotected;  and  2)  large 
spalled  coating  pieces  may  pose  additional  danger 
during  engine  operation.  Turbine  blades  with  failed 
TBC’s  must  be  recoated  or  replaced.  To  help 
minimize  spallation,  a  nickel  alloy  bond  coat  is 
deposited  on  the  superalloy  substrate  before  the  TBC 
is  applied.  This  bond  coat  can  promote 
adhesion,  serve  as  a  graded  thermal 
expansion  layer,  and  limit  hot  oxidation/ 
corrosion  resulting  from  high  temperature 
oxygen  diffusion  through  zirconia.  As  a 
result  of  the  TBC’s  transparency  to  high 
temperature  oxygen  diffusion,  most  bond 
coat  alloys  in  current  production  form  a 
thermally  grown  oxide  layer  of  primarily 
Al203.  The  thickening  of  the  thermally 
grown  oxide  with  repeated  thermal  cycling 
is  accompanied  with  increased  spallation.  Research, 
sponsored  by  AFOSR,  is  being  conducted  to  suggest 
chemical  modifications  to  the  bond  coat  alloy  that  will 
improve  adhesion  at  the  ceramic/metal  interface.  It 
will  inhibit  oxide  formation  or  form  oxide  products 
more  ideally  suited  to  the  imposed  interface  and 
thermal  cycling  requirements.  In  addition  to  jet 
engine  turbine  applications,  improved  TBCs  might 
permit  more  efficient  running  of  stationary  power 
plants — providing  benefit  both  by  decreasing 
pollution  and  cost. 

Trial  and  Error  Methods 

The  physical  and  chemical  complexity  of  TBCs  has 
required  “trial  and  error”  empirical  engineering 
methods  for  most  material  advances.  From  a 


For  years,  the  TBC  of  choice  has  been 
zirconia  with  ~7-8%  yttria  (yttria-stabilized 
zirconia,  YSZ)  added  to  stabilize  the 
tetragonal/cubic  phases  of  zirconia  over 
the  relevant  temperature  cycling  range. 

This  Y203-Zr02  coating  affords  significant 
thermal  protection,  exhibiting  very  low 
thermal  conductivity.  Unfortunately,  it  is 
not  well  suited  to  oxidative/corrosive 
protection.  Furthermore,  its  materials  properties  are 
not  optimally  matched  to  the  single  crystal  nickel 
alloy  substrate.  As  a  result,  an  intermediate 
protective  layer,  commonly  called  the  “bond  coat” 
and  typically  consisting  of  a  NiCrAlY  alloy,  is 
deposited  on  the  nickel  superalloy  prior  to  deposition 
of  the  YSZ.  This  layer  provides  an  intermediate 
thermal  expansion  coefficient  to  that  of  the  superalloy 
and  the  YSZ.  It  also  is  designed  to  promote  YSZ 
adhesion,  and  must  supply  a  source  of  oxidative/ 
corrosive  protection  from  the  harsh  combustion 
gases.  The  Al  content  in  the  bond  coat  is 
responsible  for  this  latter  task.  The  oxidation  product 
of  aluminum,  Al203,  forms  a  nearly  continuous  layer 
that  serves  as  a  slow-growing,  effective  barrier 
against  further  oxidation  of  the  underlying  metal  alloy. 


"Only  with  the  recent 
advent  of  high 
performance  computing 
capabilities  is  it  feasible 
to  explore  the  detailed 
interactions  at  metal- 
ceramic  interfaces.” 


-39- 


-  Project  and  Sustain  US  Forces 


theoretical  standpoint,  the  rich  physics  underlying  the 
fundamental  nature  of  metal/oxide  interfaces  is  only 
recently  being  explored.  Before  this,  the  strong 
attractive  force  at  nonreactive  metal/ceramic 
interfaces  had  been  attributed  to  classical 
electrostatic  effects.  Although  it  is  likely  these 
electrostatic  forces  are  important,  a  theory 
emphasizing  these  contributions  to  the  exclusion  of 
chemical  bonding  mechanisms  provides  an 
incomplete  picture  of  interface  adhesion.  More 
importantly,  the  logical  conclusions  that  can  be 
inferred  from  such  a  theory  result  in  counter¬ 
productive  materials  design  predictions. 

Exploring  Metal-Ceramic  Interfaces 

Only  with  the  recent  advent  of  high  performance 
computing  capabilities  is  it  feasible  to  explore  the 
detailed  interactions  at  metal-ceramic  interfaces.  In 
many  instances,  quantum  mechanical  effects  may 
dominate  the  interface  adhesion.  Treating  the 
quantum  mechanical  problem  in  metal/ceramic 
interface  studies  is  very  computationally  challenging, 
requiring  extensive  computing  power  and  memory. 
Neglecting  to  treat  the  quantum  mechanical  problem, 
or  treating  these  effects  at  a  very  approximate  level, 
is  far  less  computationally  expensive  but  excludes 
the  effects  necessary  for  crucial  insights  into 
materials  advancements. 

Researchers  at  the  University  of  California,  Los 
Angeles,  employ  density  functional  theory  (using 
planewave  bases  and  pseudopotentials  within  the 
Vienna  Ab  Initio  Simulation  Package)  to  investigate 
atomic-level  interactions  at  ideal  ceramic/metal 
interfaces  designed  for  TBC  applications.  These 
calculations  are  computationally  intensive  but  can  be 
run  efficiently  on  parallel  platforms.  Although  the 
system  size  is  limited  to  a  few  hundred  atoms  due  to 
computational  size  constraints,  the  three-dimensional 
(3-D)  periodic  boundary  conditions  simulate  an 
effective  “infinite”  interface  neglecting  long-ranged 
relaxation  effects.  They  have  performed  both  static 
calculations,  where  the  ionic  coordinates  are  relaxed 
to  a  local  minimum  energy  configuration  using  a 
conjugate-gradient  algorithm,  and  molecular 
dynamics  simulations  of  high-temperature  annealing 
to  better  approximate  the  extreme  environment  in  jet 
engine  combustion  chambers. 

Previously,  the  calculations  provided  insight  into 
atomic-level  contributions  to  the  observed  spallation 
of  TBCs.  They  found  the  ideal  interface  between 
Al203  and  the  bond  coat  alloy,  as  well  as  the  Al203 
interface  with  the  zirconia  topcoat,  is  unfavorable 
from  the  standpoint  of  interfacial  adhesion  and 
chemical  bonding.  In  particular,  the  ceramic/metal 


interface  adhesion  dramatically  decreases  with 
increasing  oxide  thickness.  Their  calculations 
indicated  a  fundamental  weakness  of  such  interfaces 
could  be  linked  to  the  high  ionicity,  and  thus  closed- 
shell-like  behavior,  of  the  Al203  interface  with  an  alloy 
primarily  composed  of  Ni,  an  element  with  a  mostly 
filled  d-shell.  Such  electronic  structure  results  in 
effective  “closed-shell  repulsions”  at  the  interface 
instead  of  strong  bonding  interactions. 

Investigating  Chemical  Modifications 

Accordingly,  the  team  investigated  chemical 
modifications  to  the  metal  alloy  bond  coat  designed 
to  promote  a  strong  interface  by  permitting  bonding 
interactions  at  the  ceramic/metal  rather  than  closed- 
shell  repulsions.  They  explored  the  atomic-level 
impact  of  early  transition  metal  dopants  at  the 
ceramic/metal  interface  currently  relevant  in  TBCs 
Al203/Ni  (Figure  1).  Their  initial  results  indicated  that 
early  transition  metal  doping  of  the  bond  coat  alloy 
effect  a  dramatic  increase  in  adhesion.  Their 
calculations  showed  this  bond  coat  alloy  “doping” 
resulted  in  a  —45%  increased  adhesion,  relative  to 

nickel,  at  the  ZrCX/Ni 
interface  and  a  dramatic 
100%  increase  in 
adhesion  at  the  Al203/Ni 
interface. 

Over  the  past  year,  the 
team  has  performed 
detailed  studies 
exploring  how  the 
electron  density  is 
modified  as  a  result  of 
early  transition  metal 
interface  doping.  They 
also  investigated  the 
effects  of  modifying  the 
interface  electronic 
structure  through 
altering  the  oxygen 
states  of  the  ceramic. 
They  calculated  the  ideal  adhesion  at  the  Si02/Ni  and 
Zr02/Si02  interfaces  and  found  it  to  be  much  greater 
than  the  adhesion  at  the  corresponding  interfaces 
with  Al203  instead  of  Si02.  Similar  to  early  transition 
metal  interface  dopants,  the  dramatically  increased 
adhesion  is  a  result  of  quantum  mechanical 
interactions  at  the  interface.  The  Si-0  bond  in  Si02  is 
less  ionic  than  the  AI-0  bond  in  Al203,  thus  the 
oxygen  in  Si02  does  not  contribute  local  interface 
repulsions  like  those  seen  in  the  Al203/Ni  couple. 
Local  bonding  interactions  are  essential;  using  only 
electrostatic  considerations  as  the  means  to  predict 


Ceraaic 

Interface 
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Figure  1 .  This  figure  displays 
the  doped  metal-ceramic 
interface  structure  from  atomic- 
level  calculations.  Although  the 
actual  number  of  atoms  in  the 
simulation  is  fairly  small, 
periodic  boundary  conditions 
allow  the  team  to  simulate  an 
“infinite’’  interface. 
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Figure  2.  Contour  plot  of  the  electron  density  difference  between  the 
sum  of  the  isolated  metal  substrate  and  ceramic  coating  densities 
and  the  metal-ceramic  interface  (Al203/Ti-Ni)  electron  density.  The 
color  scale  runs  from  red  to  violet  with  red  showing  increased  density 
in  the  interface  and  violet  indicating  decreased  density.  The  yellow 
“circles"  in  the  horizontal  region  indicated  by  the  blue  line  display  a 
region  of  metal-metal  bonding  as  a  result  of  interface  formation. 


strong  metal-ceramic  interfaces  for  TBC  applications 
results  in  the  wrong  physics.  Their  calculations 
indicate  chemical  bonding  effects  are  responsible  for 
strong  heterogeneous  interface  adhesion. 

Predicting  Materials  Improvements 

Previous  studies  have  concentrated  on  achieving  a 
detailed  understanding  of  atomic-level  weaknesses  at 
idealized  interfaces  related  to  those  found  in  TBCs 
and  predicting  materials  improvements  to  these 
interfaces.  The  team’s  predictions  that  quantum 
mechanical  effects  play  the  dominant  role  in  strong 
interface  adhesion  may  allow  new  materials  design 
paradigms  that  permit  informed  engineering 
modifications  to  real  TBCs.  Predictions  for  improving 
materials  performance  should  significantly  decrease 
the  “trial  and  error”  phase  of  materials  design  for 
these  applications. 

Although  current  results  may  provide  fruitful  avenues 
for  materials  improvements,  additional  challenges 
remain.  In  the  future,  the  team  plans  to  study  the 
effect  of  diffusion  and  segregation  of  harmful  dopants 
to  the  metal  alloy/oxide  interface.  A  detailed 
understanding  of  how  interface  adhesion  is  reduced 
by  undesirable  segregants,  such  as  sulfur,  may 
suggest  means  by  which  the  detrimental  impact  of 
such  elements  could  be  limited  or  eliminated. 
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The  Army  needs  lightweight,  compact,  fuel- 
efficient  propulsion  systems  for  helicopters 
and  tanks  for  the  Army  After  Next.  This 
will  require  significant  improvement  in  the 
performance  of  the  next  generation  gas  turbine 
engines.  Reduction  in  engine  size  and  weight  will 
also  reduce  the  logistics  burden  for  rapidly 
deployable  forces.  Reduced  fuel  consumption  is 
especially  critical  since  fuel  is  a  major  logistics 
burden.  Under  the  leadership  of  the  Army  Research 
Office  (ARO),  scientists  are  developing  advanced 
fuel-efficient  systems  to  reduce  fuel  consumption 
without  sacrificing  performance.  However, 
experimental  investigations  of  such  systems  are 
complicated  by  the  difficulty  in  obtaining  non- 
intrusive  access  to  the  hot,  turbulent  reacting  region 
in  the  combustor.  Numerical  tools  that  can  predict 
the  unsteady  combustion  processes  may  be  able  to 
provide  a  new,  previously  unavailable  capability  to 
investigate  the  mixing  enhancement  process.  This, 
in  turn,  will  advance  the  design  process. 

Revolutionizing  the  Design  of  Fuel 
Injectors 

The  department’s  engineers  and  scientific 
researchers  must  revolutionize  the  design  of  fuel 
injectors  to  increase  fuel  efficiency  and  reduce  the 
environmental  impact  of  combustion-powered 
vehicles.  Fuel  injectors  are  designed  to  rapidly  and 
thoroughly  mix  fuel  with  the  surrounding  air  (oxidizer) 
before  the  mixture  is  burnt.  Ineffective  or  incomplete 
mixing  produces  sooty  flames  and  heat  loss,  which 
causes  a  substantial  performance  drop.  Traditional 
injectors  are  designed  with  circular  or  round  jets 
since  they  generally  deliver  a  symmetric  (though  not 
necessarily  uniform)  fuel  mixture.  Geometries  such 
as  square  jets  (Figure  1),  however,  may  offer 
increased  mixing  rates.  Fuel  injectors  that  employ 
active  control  of  fuel  dispersion  may  offer  even  more 
enhanced  mixing  efficiency.  For  example,  synthetic 


microjet  forcing  of  fuel  injectors  has  shown  a 
remarkable  ability  to  increase  and  control  mixing. 
Synthetic  jets  do  not  add  additional  fuel  to  the 
mixture  but,  instead,  work  with  fluid  already  injected 
into  the  mixing  environment. 

Synthetic  jets  can  be  used  to 
increase  fuel/air  mixing  rates 
and/or  to  actively  control  the 
primary  jet  flow  or  boundary 
layers  by  periodically 
sucking  and  blowing  fluid 
into  and  out  of  their  micro¬ 
scale  storage  chambers,  as 
shown  in  Figure  2. 

Innovative  techniques  based 
on  micro-electro- 
mechanical  systems  (MEMS) 
technology  are  currently  being  studied  where  the  fuel 
jet  is  actively  forced  using  embedded  synthetic 
micro-jets.  Significant  enhancement  of  fuel-air 
mixing  appears  to  be  occurring  due  to  these  MEMS 


Figure  1.  Simulation  of 
square  jet  using  Lattice- 
Boltzman  method.  Square 
vortices,  depicted  by  the 
vorticity  magnitude 
isosurface,  are  propelled 
downstream  (upwards)  with 
the  flow.  The  streamlines  are 
drawn  from  outside  the  jet 
core  reveal  the  high  mass 
entrainment  rates 
characteristic  of  square  jets. 


“Numerical  tools  that  can 
predict  the  unsteady 
combustion  processes 
may  be  able  to  provide  a 
new,  previously 
unavailable  capability  to 
investigate  the  mixing 
enhancement  process. 
This,  in  turn,  will  advance 
the  design  process." 
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Figure  2.  Synthetic  microjets 
are  used  to  increase  fuel/air 
mixing  and  boundary  layer 
control.  Outside  air  is  sucked 
into  the  jet  cavity  and  then 
rapidly  expelled  outwards  into 
the  surround  mixture.  Shown 
here,  is  the  simulation  of  a 
high-aspect-ratio  (high  width 
to  length  ratio)  synthetic  jet. 


devices,  however,  detailed  observation  of  the  flow 
field  generated  by  these  micro-scale  actuators  is 
difficult,  if  not  impossible. 

High  Performance  Modeling  of 
Synthetic  Jets 

Modeling  of  fuel  injectors  is  a  challenging  task  due  to 
their  small  size.  The  small  dimensions  restrict  the 
maximum  allowable  time-step  that  may  be  taken  in  a 
traditional  CFD  simulation.  Time-accurate  (unsteady) 
simulations  are  required  to  capture  the  dynamics  of 
the  unsteady  nature  of  jet  motion  and  fuel/air  mixing. 
The  restrictive  time-step  effect  is  even  more 
pronounced  for  micro-scale  synthetic  jets,  which 
have  to  be  small  enough  to  actually  fit  inside  the 
primary  fuel  jets.  A  new  modeling  technique,  Lattice- 
Boltzman  Equation  (LBE),  is  used  to  simulate  the 
small-scale  flow  in  one  synthetic  jet  and  other  novel 
fuel  injector  designs.  Unlike  traditional  CFD,  which 
represents  the  working  fluid  as  a  continuous  system 
(macroscopic  view),  LBE  treats  the  fluid  as  a 
collection  of  particles  governed  by  a  single  statistical 
distribution  equation  (microscopic  view).  This 
technique  is  advantageous  when  simulating  small- 
scale  fluids  since  its  time-step  limitations  are  not 
restrictive.  Additionally,  LBE  is  easily  parallelized  and 
is  highly  scalable  on  a  parallel  high-performance 
computing  system. 

In  the  current  studies,  various  3-D  non-circular  and 
synthetic  jets  are  simulated  using  the  LBE  method. 
The  objective  of  this  study  is  to  investigate  the 
turbulent  dynamics  of  these  jets  and  to  evaluate  the 
effectiveness  and  accuracy  of  the  3-D,  parallel  LBE 
method.  Typically,  2-3  million  grid  points  are  used  to 
resolve  the  resulting  unsteady  flow.  This  complexity 


requires  600  CPU  hours  and  nearly  2  gigabytes  of 
memory  on  a  distributed  shared-memory  parallel 
machine.  The  extreme  efficiency  of  the  LBE 
algorithm  coupled  with  the  high-performance 
computing  resources  available  at  the  DoD  MSRCs 
allows  high-resolution  parametric  jet  simulations  with 
rapid  turn-around  times. 

Excellent  results  have  been  obtained  from  the  LBE 
simulations.  Simulations  of  free  and  forced  square 
jets  have  been  able  to  capture  phenomena  such  as 
axis  switching  (vortices  inverting  as  a  result  of  the 
non-uniform  flow)  and  increased  mass  entrainment. 
Similarly,  direct  comparisons  with  experiments  on  a 
series  of  synthetic  jets  have  also  been  conducted. 
The  simulations  show  good  agreement  with  the 
experiments,  adding  credibility  to  the  numerical 
method.  Finally,  the  interaction  of  a  square  jet  with  a 
steady  cross-flow  (as  would  be  used  for  boundary- 
layer  control  or  combustor  wall  cooling)  has  been 
investigated.  Complex  flow  structures  such  as 
“horse-shoe”  and  “hairpin”  vortices  have  been 
observed. 

Several  extensions  to  the  LBE  method  are  currently 
underway.  These  include  extending  this  promising 
method  to  combustion  (which  requires  the  modeling 
of  chemical  reactions  and  heat-release)  and  coupling 
the  LBE  simulations  to  Large-Eddy  Simulations 
(LES).  LES  is  an  effective  tool  for  simulating 
unsteady,  complex  flows  such  as  those  found  inside 
gas  turbine  combustors.  The  coupled  LBE/LES 
method  would  allow  the  simulation  of  both  the  fuel 
injectors  mixing  regions  and  the  combustor  at  an 
affordable  computational  cost.  The  combined  LBE/ 
LES  approach  provides  a  unique  simulation 
capability  that  can  be  used  to  investigate  the 
performance  of  these  fuel  injectors  under  realistic 
operating  conditions.  In  the  future,  many  other 
issues  of  practical  interest  to  DoD  such  as  high- 
pressure  combustion,  supercritical  fuels,  and 
combustion  instability  can  be  addressed  with  these 
codes. 
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The  United  States  Army  is  developing 

advanced  lightweight  armor  to  increase 
vehicle  protection  and  reduce  vehicle 
weight.  One  class  of  materials  that  has  been 
considered  for  armor  applications  in  recent  years  is 
ceramics.  Ceramic  materials  are  very  strong  in 
compression,  relatively  weak  in  tension,  and  tend  to 
be  lightweight  when  compared  to  more  traditional 
armor  materials  such  as  metals.  Certain  ceramics 
have  also  exhibited  a  phenomenon  called  “ceramic 
dwell”.  Ceramic  dwell  occurs  when  a  high  velocity 
projectile  impacts  a  ceramic  target  and  then  dwells 
on  the  surface  of  the  ceramic  with  no  significant 
penetration.  These  characteristics  make  ceramics 
well  suited  for  armor  applications,  but  also  present  a 
significant  design  challenge  when  incorporated  into 
armor  systems. 

Ballistic  Testing  Used  to  Develop 
Ceramic  Armor 

Historically,  ceramic  armor  has  been  developed  by 
rigorous  use  of  ballistic  testing.  Although  much 
advancement  has  been  made  in  ceramic  armor 
design  primarily  using  ballistic  testing,  it  is  a  time 
consuming  and  very  expensive  process.  A  better 
approach  is  to  use  ballistic  testing  along  with 
computations.  Effective  armor  design  can  be 
enhanced,  cost  can  be  reduced,  and  the  behavior 
better  understood  by  the  use  of  HPC  resources  if  the 
computational  models  adequately  represent  the 
material  behavior.  Furthermore,  the  HPC  systems 
have  the  memory  and  speed  required  to  perform 
these  computations  in  a  timely  and  efficient  manner. 

Recent  advancements  in  the  development  of  ceramic 
material  models  and  numerical  algorithms  at 
AHPCRC,  allows  for  accurate  representation  of 
ceramic  material  behavior.  Specifically,  the  ability  to 
model  the  phenomenon  of  ceramic  dwell  has  been 
developed  and  computations  have  been  performed 
to  investigate  ceramic  dwell  and  how  it  is  affected  by 
target  geometry. 


Figure  1  presents  the  computational  results  for  three 
target  configurations.  All  three  targets  are  comprised 
of  a  ceramic  core  surrounded  by  metal.  The  baseline 
geometry  is  a  configuration  that  was  investigated 
experimentally.  The  computational  results  accurately 
represent  the  behavior  exhibited  in  the  experiment. 
The  other  two  target  configurations,  slight 
modifications  of  the  baseline  geometry,  were 
performed  to  investigate  the  effect  of  diameter  and 
cover  plate  attachment.  The  penetration  results  are 
presented  as  a  function  of 
time  at  the  bottom  of 
Figure  1.  The  results 
indicate  that  the  best 
performing  design  is  the 
large  diameter 
configuration  (least  amount 
of  ceramic  penetration  at 
60  microseconds)  and  the 
worst  performing  design  is 
the  detached  cover  (most 
amount  of  ceramic 
penetration  at  60 
microseconds). 

Computations  such  as 
these  can  be  performed  to 
provide  insight  into  the 

complex  behavior  of  ceramic  armor  and  aid  the 
armor  designer  in  developing  improved  armor 
systems. 

The  ability  to  accurately  simulate  the  ceramic  dwell 
phenomenon  allows  the  Army  reseachers  to  design 
ceramic  armor  in  an  efficient,  cost  effective  manner. 
This  new  capability  is  currently  being  used  by  the 
ARL  and  the  Tank  Automotive  Research, 

Development  and  Engineering  Center  (TARDEC)  to 
help  design  ceramic  armor  for  the  Future  Combat 
System  (FCS). 


"Effective  armor  design 
can  be  enhanced,  cost  can 
be  reduced,  and  the 
behavior  better  understood 
by  the  use  of  HPC 
resources  if  the 
computational  models 
adequately  represent  the 
material  behavior. 
Furthermore,  the  HPC 
systems  have  the  memory 
and  speed  required  to 
perform  these 
computations  in  a  timely 
and  efficient  manner.” 
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Figure  1 .  Computational  results  for  a  baseline  ceramic  target 
geometry,  a  large  diameter  geometry  and  a  detached  cover 
geometry.  For  all  three  geometries,  the  ceramic  material  is 
encapsulated  by  metal.  Also  shown  are  the  ceramic  penetration  time 
histories. 
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As  composite  materials  are  increasingly 
used  in  weapon  and  combat  systems 
(both  retrofitted  and  new  components),  it 
is  increasingly  important  to  fully  understand 
the  complexities  associated  with  their  manufacture. 
Structural  materials  used  in  the  past,  such  as  metals, 
have  well-established  design,  manufacturing,  and 
testing  protocols.  The  same  cannot  be  said  of 
composites.  Even  after  years  of  use,  the  structures 
remain  risky  in  terms  of  cost  and  difficulties 
associated  with  manufacture.  Traditionally, 
experienced  process  engineers  have  manufactured 
these  parts  using  “best  guess”  approaches. 

However,  the  complexities  of  the  new  designs  being 
developed  to  support  FCS  significantly  highlight  the 
limits  of  that  approach.  Composites  with  embedded 
structures,  for  example,  significantly  increase  the 
level  of  difficulty  in  producing  a  viable  part  on  the  first 
manufacturing  attempt. 

Modeling  Composites  and  the 
Manufacturing  Process 

The  best  solution  to  overcome  these  problems  is  a 
physics-based  modeling  and  simulation  approach. 
Liquid  injection  molding  of  composites  usually 
involves  some  fiberglass  or  organic  matrix  fiber 
structure  being  placed  into  a  mold  and  being 
injected  with  a  thermosetting  resin.  Darcy’s  law 
governs  the  flow  of  resin  through  a  porous  media.  It 
relates  the  velocity  field  to  the  pressure  gradient 
through  fluid  velocity  and  the  fiber  preform 
permeability.  Darcy’s  law  is  defined  as 


n 


where  u  is  the  velocity  field,  p  is  the  fluid  viscosity, 
and  P  is  the  pressure.  Permeability  K  is  a  measure  of 
the  resistance  experienced  by  a  fluid  as  it  flows 
through  a  porous  media  and  is  an  important  material 
characteristic  involved  in  the  flow  simulations. 
Complex  structures  are  discretized  using  finite 
element  meshes  to  model  the 
geometric  complexities. 

Material  regions  in  the  model 
are  defined  based  on  the 
permeability  characteristics  of 
the  fiber  preforms. 

Traditional  numerical 
approaches  to  model  the 
manufacturing  phenomena, 
such  as  the  finite  element- 
control  volume  method,  were 
considered  and  found  inadequate.  The  stability 
conditions  associated  with  the  solution  required 
computer  times  exceeding  weeks  of  single  processor 
or  parallel  computer  time.  However,  the  newer 
methodologies,  found  in  the  Composite 
Manufacturing  PrOcess  Simulation  Environment 
(COMPOSE)  suite  of  tools,  were  developed  with  a 
focus  on  parallelism  and  fundamental  algorithmic 
changes.  Now,  large-scale  process  simulations  of 
even  the  most  detailed  and  complex  parts  are 
possible  with  results  available  in  hours  rather  than 
weeks  or  even  months.  Process  engineers  can 
perform  parametric  studies  to  identify  potential 
manufacturing  problems  and  optimize  the 
manufacturing  process  before  capital  investment  in 
mold  tooling  is  made.  COMPOSE  developers  are 
working  to  expand  these  capabilities  by  adding 
sensitivity  analysis  and  thermal  and  cure  modeling  to 
the  parallel  solutions.  Recent  additions  such  as  a 
cross-platform  Java-based  user  interface  are  putting 


“Now,  large-scale 
process  simulations  of 
even  the  most  detailed 
and  complex  parts  are 
possible  with  results 
available  in  hours 
rather  than  weeks  or 
even  months." 
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the  technology  in  the  hands  of  process  engineers 
and  manufacturers  even  faster. 

COMPOSE  has  been  used  to  model  numerous 
geometries  found  in  rotorcraft  applications.  These 
parts  include  fuselage  components,  stabilizers,  and 
interior  components.  Recent  work  with  Sikorsky 
United  Technologies  was  directed  toward 
manufacturing  improvements  for  composite  seat 
fittings  found  in  the  RAH-66  Comanche  helicopter 
(shown  in  Figure  1).  This  part,  shown  in  Figure  2,  is 
thin-shelled,  made  from  a  graphite  woven  fiber,  and 
injected  with  a  polymer  resin.  By  using  computer 
modeling,  it  was  possible  to  verify  the  injection 
scheme  anticipated  in  production  and  to  suggest 
improvements.  Process  simulations  were  conducted 
using  various  numbers  of  processors  (Figure  3 
shows  a  16-processor  partition).  Vent  locations, 
optimally  placed  near  areas  that  are  last  to  fill,  were 
also  determined  from  the  model.  Detailed  parametric 
studies  and  numerous  white  papers  addressing  the 
manufacturing  practices  of  this  part  were  possible 
thanks  to  the  quick  turn-around  time  provided  by  the 
parallel  COMPOSE  suite  of  tools.  One  example 
showing  resin  flow  progression  through  the  part, 
based  on  injection  into  a  channel  along  the  bottom 
edge,  is  shown  in  Figure  4. 


Transitioning  to  DoD  Industrial 
Partners 

This  work  is  part  of  a  larger  effort  directed  towards 
facilitating  and  enhancing  the  use  of  composite 
materials  in  military  systems.  ARL  and  the 
Universities  of  New  Orleans  and  Minnesota  are 


conducting  the  basic  research  behind  process 
modeling.  Transition  to  DoD  industrial  partners  such 
as  Boeing  and  Sikorsky  is  being  conducted  through 
the  Aviation  Applied  Technology  Directorate  (AATD), 
Weapons  and  Materials  Research  Directorate-ARL, 
and  the  US  Army  Aviation  and  Missile  Command. 
These  process  models  are  one  of  the  significant 
manufacturing  technology  advancements  found  in 
the  Rotary  Wing  Structures  Technology 
Demonstration  project  managed  by  AATD. 
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Cavitation  is  “the  formation  of  partial 

vacuums  in  a  liquid  by  a  swiftly  moving 
solid  body  (as  a  propeller)  or  by  high- 
intensity  sound  waves.”  1  In  liquid 
handling  systems  like  turbomachinery  pumps  and 
propellers,  low  pressure  fluctuations  downstream  of 
the  rotor  can  induce  cavitation,  leading  to  acoustic 

noise,  loss  of  performance, 
and  structural  damage.  This 
complex,  highly  unsteady 
flow  phenomenon  is  poorly 
understood  and  cannot  be 
predicted  by  the  traditional 
CFD  methods.  In  the 
present  work,  sponsored  by 
the  Navy,  researchers 
employ  LES  to  study  the 
flow  past  a  stator 
(stationary)-rotor 
combination  with  tip 
leakage,  with  the  objective  of  understanding, 
predicting,  and  eventually  controlling  cavitation.  The 
LES  code  developed  for  this  project  has  the  potential 
to  be  a  useful  research  and  design  tool  for  other 
future  weapons  systems  design  efforts. 

Understanding  the  Underlying 
Causes  of  Cavitation 

Traditionally,  scientists  used  experiments  to  study  tip- 
leakage  flow  in  turbomachines.  There  is  evidence 
suggesting  that  low-pressure  events  and  cavitations 
are  strongly  related  to  the  turbulence  associated  with 
the  tip-leakage  vortex  near  the  endwall  (Figure  1). 
However,  knowledge  of  the  detailed  mechanisms  is 


1  Merriam  Websters  Collegiate®  Dictionary 


still  quite  limited.  Computational  studies  of  this  flow 
have  customarily  relied  upon  Reynolds  Averaged 
Navier-Stokes  (RANS)  models,  which  cannot 
describe  the  highly  unsteady  flow  and  pressure 
characteristics.  In  addition,  the  tip-gap  region  has 
presented  CFD  practitioners  with  considerable 
challenges  in  terms  of  grid  topology  and  resolution. 

Introduction  of  the  Large-Eddy 
Simulation 

The  HPCMP  resources  allow  scientists  to  use  LES, 
which  is  computationally  much  more  intensive  than 
RANS  calculations.  As  shown  in  Figure  2,  the 
present  LES  research  attempts  to  resolve  a  wide 
range  of  flow  scales  associated  with  the  stator  wake, 
the  rotor  tip-leakage  vortex,  and  the  turbulent 
boundary  layers.  The  computational  domain  is  quite 
long  since  the  region  of  interest  extends  several 
chord  lengths  into  the  wake.  The  flow  is  three- 
dimensional  without  a  statistically  homogeneous 
direction,  requiring  extremely  long  sampling  time  to 
obtain  converged  flow  statistics.  A  full  simulation 
requires  more  than  25  million  grid  points  and  100,000 
time  steps.  A  typical  production  run  requires  over 
50,000  single-node  CPU  hours  and  10  gigabyte  (GB) 


“The  simulations 
conducted  to  date  have 
shown  encouraging 
results  in  terms  of 
predicting  both  the 
qualitative  and 
quantitative  features  of 
the  rotor  tip-leakage  flow 
as  observed 
experimentally.” 
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inflow 

Figure  2.  Instantaneous  streamwise  velocity  iso-surfaces  revealing 
the  tip-leakage  flow  and  rotor-blade  wake 


of  memory  on  the  SGI  Origin  3800  and  generates 
over  50  GB  of  data,  rendering  these  simulations 
feasible  only  on  large,  high  performance,  parallel 
platforms.  Large-scale  simulations  of  this  magnitude 
would  have  been  impossible  without  HPCMP 
resources. 

A  large-eddy  simulation  code,  which  combines  an 
immersed-boundary  technique  with  a  generalized 
curvilinear  structured  grid,  has  been  employed  to 
carry  out  the  simulations.  From  the  preliminary 
results,  mean  velocities,  Reynolds  stresses  and 
energy  spectra  show  reasonable  agreements  with 
experimental  measurements  (Figure  3).  Theflowfield 
exhibits  strong  circular  motions  associated  with  the 
tip-leakage  and  tip-separation  vortices.  These 
swirling  structures  convect  downstream,  expand  in 
size  and  generate  intense  turbulent  fluctuations  in  the 
endwall  region.  Strong  pressure  fluctuations  that 
may  be  responsible  for  the  cavitation  and  acoustic 
noise  have  also  been  observed.  Analysis  of  the 
results  indicates  that  further  improvement  may  be 
obtained  by  increasing  the  streamwise  resolution  in 
the  region  where  the  tip-leakage  and  trailing  edge 
vortices  interact  with  each  other. 

Expanding  Naval  Understanding 

The  simulations  conducted  to  date  have  shown 
encouraging  results  in  terms  of  predicting  both  the 
qualitative  and  quantitative  features  of  the  rotor  tip- 
leakage  flow  as  observed  experimentally.  Grid- 
refinement  studies  are  underway,  and  the  solutions 
will  be  fully  validated.  Once  this  is  done,  Professor 
Moin’s  team  will  focus  on  a  detailed  analysis  of  the 
data  with  emphasis  on  understanding  the  turbulence 
characteristics,  vortex  dynamics  and  pressure 
fluctuations  in  and  downstream  of  the  tip-gap  region. 
In  addition,  stator-rotor  interaction  and  rotational 


* 
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Figure  3.  Energy  spectra  of 
each  velocity  component  as  a 
function  of  frequency  at  a 
location  in  the  tip-leakage 
region 


effects  will  be  included  to  simulate  the  flow  in  a  more 
realistic  configuration.  The  parallel  performance  of 
the  code  will  be  further  enhanced  by  employing 
domain  decomposition  strategies  coupled  with 
explicit  Message  Passing  Interface  (MPI),  instead  of 
the  loop  level  directives  (OpenMP)  currently  in  use. 

The  results  of  this  research  will  eventually  lead  to  a 
new  generation  of  quieter  weapons  systems  capable 
of  greater  performance  at  reduced  costs. 
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Spoil  the  Strikes 

M.S.  Jurkovich,  T.E.  Rittinger,  and  R.M.  Weyer 

Aeronautical  Systems  Center  (ASC),  Wright-Patterson  Air  Force  Base,  OH 


The  Air  Force  has  recently  undergone  a 
second  round  of  flight  tests  to  address 
problems  identified  on  a  new  refueling  pod 
for  the  KC-1 35R  aircraft.  It  is  a  hose  and 
drogue  system  called  the  Multi-Point  Refueling 
System  (MPRS)  (Figure  1).  When  the  hose  and 
drogue  on  the  KC-135R  are  retracted,  the  drogue 
often  strikes  and  occasionally  damages  the  wing. 

This  can  lead  to  unacceptable  downtime  to  repair  the 
wing  trailing  edge  and  tip. 


Figure  1 .  Aircraft  view  showing  installation  of  MPRS 


Conventional  Testing  Fails  to  Yield 
Solutions 

Prior  to  flight  test,  a  team  from  the  Engineering 
Directorate  of  the  Aeronautical  Systems  Center  (ASC) 
was  called  on  to  perform  CFD  modeling  on  the 
system.  Previous  wind  tunnel  and  flight  tests  failed  to 
identify  the  source  of  the  problem.  The  wind  tunnel 
measurements  were  not  taken  close  enough  to  the 
wing  to  discover  the  flow  phenomena  responsible  for 
causing  the  drogue  to  strike  the  wing. 

High  Performance  Computing 
Solutions 

The  ASC  research  team  used  HPCMP  computational 
resources  and  CFD  to  solve  multiple  flight  conditions 
in  a  reasonable  amount  of  time.  The  investigating 
team  used  CHSSI  developed  code,  Cobalt60,  for  the 
CFD  solutions.  They  coupled  it  with  an  in-house, 


steady-state  catenary  solver.  The  computational 
solutions  provided  the  entire  flowfield,  allowing 
engineers  to  identify  sources  of  the  problem. 

Solution  visualization  using  Fieldview  identified  a 
large  upwash  in  the  flowfield  just  behind  the  MPRS 
refueling  pod.  As  Figure  2 
illustrates,  and  as  calculated 
by  the  catenary  solver,  the 
drogue  was  being  pushed  up 
into  the  wing  by  the  flowfield  in 
proximity  to  the  pod.  The 
design  of  the  back  end  of  the 
pod  allows  the  flow  to  remain 
attached  and  turn  upward. 

This  flow  behind  the  pod 
contributes  to  the  general 
upwash  flowfield  created  by 
the  wing  tip  vortex.  HPCMP 
resources  allowed  ASC  to  compute  many  variations 
on  the  geometry  including  the  baseline  geometry,  the 
French  Air  Force  variant,  the  wind  tunnel  geometry, 
rotated  fins,  inboard  fins,  three  spoilers,  and  the 


Figure  2.  Baseline  geometry  resulting  in  local  upwash  behind  pod 


"The  use  of  major 
shared  resource  center 
resources  is  saving 
program  dollars  in 
helping  identify  causes 
and  potential  solutions 
to  a  current  operational 
problem  that  were  not 
identifiable  by  other 
methods  at  hand.’’ 


-50- 


2002  Success  Stories 


entire  aircraft  in  sideslip.  Computational  grids  were 
generated  with  the  National  Aeronautics  and  Space 
Administration,  Langley  and  Air  Force  Research 
Laboratory  codes  Gridtool/Vgrid  and  Blacksmith. 
The  largest  grid  was  14  million  tetrahedral  and 
prismatic  cells. 

The  largest  spoiler  geometry  provided  the  best 
flowfield  analytically  (Figure  3)  and  was 
manufactured  for  flight-testing.  The  flight  test 


showed  the  spoiler  to  be  partially  successful  in 
reducing  the  frequency  of  wing  strikes.  Follow-on 
studies  are  already  in  progress  that  identify  means  to 
also  reduce  the  impact  of  the  wing  tip  generated 
upwash  on  the  retracting  drogue.  The  use  of  MSRCs 
is  saving  program  dollars  in  helping  identify  causes 
and  potential  solutions  to  a  current  operational 
problem  that  were  not  identifiable  by  other  methods 
at  hand. 

CTA:  Computational  Fluid  Dynamics 

Computer  Resources:  SGI  Origin  3000  [ARL,  ERDC], 

IBM  SP3  [MHPCC,  ASC],  and  SGI  Origin  2000  [ASC] 
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CFD  Modeling  of  Antenna  Structures  Using  Cobalt 
Internal  Boundary  Conditions 

Susan  Polsky 

Naval  Air  Warfare  Center,  Aircraft  Division  (NAWCAD),  Patuxent  River,  MD 


The  objective  of  the  Ship  Aircraft  Airwake 
Analysis  for  Enhanced  Dynamic  Interface 
(SAFEDI)  Program  is  to  develop 
computational  fluid  dynamics  methods  to 
predict  the  time-accurate  airwake  produced  by  Navy 
ships  for  application  in  manned  and  unmanned  flight 
simulation.  Wind  tunnel  tests  have  shown  that 
“small”  geometric  details  on  ships  can  significantly 
affect  the  character  of  the  airwake  turbulence  that 
aircraft  must  operate  in.  These  “small”  geometric 
features  include  the  antenna  arrays  that  are  placed 
atop  the  ships  superstructure.  These  arrays  are  very 
complex  geometrically  and  present  an  extremely 
difficult  gridding  problem  for  both  grid  construction 
and  required  grid  density. 

Difficulties  of  Modeling  Antenna 
Structures 

Traditionally,  geometric  details  of  the  scale  of  the 
antennas  have  simply  been  ignored  because  of  the 
difficulties  in  modeling  the  true  geometric  detail  of 
the  structures.  In  this  work,  internal  boundary 
conditions  have  been  developed  to  model  the 
antenna  arrays  on  a  sub-grid  scale.  The  initial 
approach  taken  was  to  modify  the  momentum 
equations  using  a  source  term  to  retard  the  flow  by 
removing  momentum,  but  still  allow  for  a  permeable 
surface.  The  “porosity”  of  the  structures  were 
specified  as  a  drag  coefficients  in  an  input  deck. 
However,  it  was  found  that  determining  the  correct 
drag  coefficient  to  use  was  more  of  an  art  than 
science.  Therefore,  the  current  approach  is  to  tag  all 
the  cells  within  the  structures  as  solid  walls.  The 
areas  where  the  antennas  exist  are  specified  in  an 
input  deck  along  with  the  general  shape  (i.e., 
cylinder,  sphere,  etc.). 


Modeling  Ship  Air  Wakes 

Ship  air  wakes  are  inherently  unsteady  and  must  be 
modeled  as  such.  In  addition,  the  ships  are  very 
large  while  details  on  the  order  of  thousandths  of 
inches  must  be  modeled  to  properly  resolve 
boundary  layers.  This 
requires  large  amounts 
of  memory  and  large 
amounts  of  computer 
time. 

Visualization  of  the 
resulting  three- 
dimensional  flow  fields  is 
an  essential  part  of  the 
development  and 
analysis  tasks  related  to 
ship  air  wake 
predictions.  State-of-the- 
art  visualization  software 
is  heavily  relied  on  for  all 
aspects  of  the  program.  In  particular,  animation  of 
the  time-varying  solutions  has  provided  insight  into 
never  before  appreciated  features  of  the  air  wake 
flow  field. 

Access  to  HPC  resources  has  enabled  the 
development  of  a  new  method  of  modeling  fine 
geometric  details  for  the  prediction  of  ship  air  wake. 

Improving  Manned  and  Unmanned 
Flight  Simulation 

The  primary  impact  is  significantly  improving  manned 
and  unmanned  flight  simulation  by  providing  realistic 
airwake  models.  Improved  simulations  can  reduce 
operating  costs  by  providing  an  alternative  method 
for  pilot  training  and  dynamic  interface  envelope 
development.  In  addition,  this  work  provides  a 
unique  opportunity  to  improve  the  general 
understanding  of  ship  airwake  aerodynamics. 


“The  primary  impact  is 
significantly  improving 
manned  and  unmanned 
flight  simulation  by 
providing  realistic  airwake 
models.  Improved 
simulations  can  reduce 
operating  costs  by 
providing  an  alternative 
method  for  pilot  training 
and  dynamic  interface 
envelope  development.” 
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Finally,  the  analysis  techniques  developed  here  can 
be  used  to  aid  in  the  design  of  the  flight  decks  and 
islands  on  new  ships  and  thereby  address  flight 
operations  at  the  very  initial  stages  of  ship  design. 

An  internal  boundary  condition  to  simulate  first  order 
effects  of  an  antenna  array  on  the  flow  field  has  been 
developed,  implemented,  and  tested.  The  method 
developed  imposes  a  solid  wall  boundary  condition 
within  the  antenna  array.  This  approach  avoids  the 
need  to  determine  the  porosity  (i.e.,  drag  coefficient) 
of  varied  geometric  shapes. 

The  several  simple  test  cases  that  were  run  prove 
that  the  cells  within  the  antenna  array  box  are  being 
found  correctly,  the  orientation  of  the  antenna  is  the 
requested  orientation  and  the  momentum  is  being 
reduced  by  the  presence  of  the  antenna  array 
boundary  condition.  The  remaining  challenges  are  to 
use  the  method  for  an  actual  antenna  geometry 
(Figure  1)  and  to  validate  against  both  CFD  and,  if 
possible,  experimental  data.  For  validation  against 
CFD  data,  the  actual  antenna  geometry  will  be 
gridded  and  run  using  the  same  CFD  code,  Cobalt. 
This  process  has  already  begun  and  some  resulting 
flow  fields  are  shown  in  Figures  2  and  3.  The  method 
itself  is  still  under  development  and  may  be  modified 
based  on  results  from  the  validation  effort. 
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Figure  2.  Supporting  grid  structure 
and  application  of  subscale  boundary 
condition  (colored  by  speed) 
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Figure  3.  Two  planar  cuts  showing 
affects  of  internal  boundary 
condition 
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Flow  Control  for  Aircraft  Weapons  Bay  Enhanced 
Acoustic  Environment 

Donald  Rizzetta 

Air  Force  Research  Laboratory  (AFRL),  Wright-Patterson  AFB,  OH 


High-speed  flows  over  open  cavities 
produce  complex  unsteady 
interactions,  which  are  characterized 
by  a  severe  acoustic  environment.  At 
flight  conditions,  such  flowfields  are  both 
broadband,  small-scale  fluctuations  typical  of 
turbulent  shear  layers,  and  discrete  resonance 
whose  frequency  and  amplitude  depend  upon  the 
cavity  geometry  and  external  flow  conditions.  While 
these  phenomena  are  of  fundamental  physical 
interest,  they  also  represent  a  number  of  significant 
concerns  for  aerospace  applications. 

Researchers  at  AFRL  are  studying  these 

phenomena,  hoping  to 
influence  aircraft  weapons 
bay  environments  (as  shown 
in  Figure  1),  where 
aerodynamic  performance  or 
stability  may  be  adversely 
affected,  structural  loading 
may  become  excessive,  and 
sensitive  instrumentation  may 
be  damaged.  Acoustic 
resonance  can  also  pose  a 
threat  to  the  safe  release  and 
accurate  delivery  of  weapons 
systems  stored  within  the 
cavity. 

Shortfalls  of  the  Traditional 
Approach 

Traditionally,  solving  the  RANS  equations  has  allowed 
simulation  of  the  flow  about  aircraft  weapons  bay 
cavities.  This  approach  precluded  a  precise 
description  of  the  turbulent  cavity  flowfield,  which 
consists  of  fine-scale  fluid  structures.  The  models, 
which  accounted  for  the  effects  of  turbulence,  were 
unreliable  and  incapable  of  representing  high- 
frequency  fluctuations  of  the  fluid  structures. 


HPC  resources,  particularly  large-scale  computing 
platforms,  are  allowing  the  investigator  to  represent, 
with  more  accuracy,  the  fundamental  properties  of 
supersonic  turbulent  aircraft  weapons  bay  cavities 
(Figures  2  and  3).  In  particular,  the  engineers  can 
use  large-eddy  simulations  to  overcome  deficiencies 
of  the  RANS  approach  and  failure  of  traditional 
turbulence  models. 

Using  Large-Eddy  Simulations  to 

Improve  Acoustic  Pressure  Levels 

Large-eddy  simulations  numerically  generated  the 
turbulent  flowfields  about  weapons  bay  cavities.  The 
ability  of  high-frequency  forced  mass  injection  to 
suppress  cavity  resonant  acoustic  modes  was 
investigated.  Scientific  visualization  was  used 
extensively  to  clarify  the  physical  characteristics  of 
unsteady  three-dimensional  turbulent  cavity 
flowfields.  The  construction  of  videos  made  it 


"High  performance 
computing  resources, 
particularly  large-scale 
computing  platforms, 
are  allowing  the 
investigator  to 
represent,  with  more 
accuracy,  the 
fundamental  properties 
of  supersonic  turbulent 
aircraft  weapons  bay 
cavities.” 
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Figure  2.  Instantaneous  Mach  Number  Contours 
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Figure  3.  Instantaneous  Vorticity  Contours 


possible  to  understand  the  basic  mechanisms 
contributing  to  the  suppression  of  resonant  acoustic 
modes  through  forced  mass  injection.  It  was  found 
that  the  use  of  flow  control  appreciably  reduced 
amplitudes  of  acoustic  pressure  levels  within  the 
cavity. 

Using  HPC  resources,  the  engineers  numerically 
reproduced  supersonic  turbulent  flow  about  an 
aircraft  weapons  bay  both  with  and  without  flow 
control.  Large-eddy  simulations  have  made  it 
possible  to  correctly  describe  the  small-scale  fluid 
structures  that  characterize  turbulence.  Pulsed  mass 
injection  was  found  to  be  an  effective  flow  control 
mechanism  for  mitigating  undesirable  resonant 
acoustic  modes.  This  considerably  enhanced  the 
acoustic  environment  within  the  weapons  bay, 
thereby  reducing  the  loading  and  potential  damage 
to  weapons  systems,  surrounding  structure,  and 
instrumentation. 
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Using  Strongly  Coupled  Fluid/Structure  Interaction 
Code  to  Predict  Fighter  Aircraft  Wing  Response  to 
Anti-Aircraft  Artillery  (AAA)  Damage 
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Sponsor:  Joint  Technical  Coordinating  Group  for  Aircraft  Survivability 


It’s  becoming  clear  that  current  static  loading 
techniques  for  live-fire  testing  fail  to  accurately 
replicate  the  loads  of  aircraft  that  are  damaged 
in  flight.  Vulnerability  assessments  based  on 
such  loading  techniques  may  also,  in  turn,  fall  short 
of  providing  accurate  and  complete  results.  The 
good  news  is  that  advances  in  a  proposed  technique 
for  dynamic  live-fire  ground  testing  may  remedy  the 
shortcomings  of  current  static  ground  testing.  High 
performance  computers  are  being  used  to  develop  a 
methodology  to  predict  the  in-flight  response  of 
manned  and  unmanned  air  vehicles  to  hostile  threats 
they  may  encounter  during  wartime  operations.  The 
methodology  gives  DoD  a  physics-based,  high 
fidelity  tool  for  use  in  new  aircraft  design  as  well  as 
pretest  prediction  and  dynamic  test  planning  of 
on-going  and  future  live-fire  test  and  evaluation. 

Shortfalls  of  Static  Ground  Testing 

In  the  past,  only  static  predictions  could  be  made 
using  engineering  level  tools,  which  neglected  the 
influence  of  structural  damage  to  the  airflow  over  the 
aircraft.  The  main  shortfall  with  static  ground  testing 
is  that  the  quasi-static  techniques  do  not  account  for 
changes  in  structural  stiffness  and  mass  that  occur 
from  damage.  As  such,  current  loading 
methodologies  fail  to  reconfigure  correctly  for 
representing  in-flight  loads.  Ground  loading 
methodologies  also  fail  to  consider  damage  induced 
changes  to  the  flutter  envelope  that  can  lead  to 
premature  failure. 


Boundary  Element  Method  Models 
Flow  Fields 

The  methodology  of  this  research  was  to  use  HPC 
resources  to  model  a  current 
fighter  aircraft  structure  using 
the  Lagrangian  perspective 
coupled  with  a  boundary 
element  method  for  modeling 
the  flow  field  around  the 
aircraft.  The  boundary  element 
method  used  in  this  work  is 
consistent  with  several 
“paneling  methods”  which  are 
used  for  inviscid, 
incompressible  flows.  The 
wing  model  was  initially  loaded 
with  the  airloads  and  g-loads 
representing  a  given  flight 
condition  (Figure  1  and  Figure 
2).  Ten  seconds  into  the  simulation,  a  section  of 
structure  was  instantaneously  removed  to  simulate 
the  damage  resulting  from  a  threat  detonation  or 
impact.  The  simulation  was  allowed  to  continue  for 
another  10  seconds  and  resulting  stresses  and  wing 
displacements  were  observed  (Figure  3). 

Simulations  of  this  type  require  stepping  in  time  with 
steps  on  the  order  of  0.1  microseconds.  Thus  a  20 
second  simulation  would  require  200  million  steps. 
Without  the  HPC  resources,  each  simulation  would 
have  taken  days  to  perform.  With  the  multi-processor 
capability  that  HPC  brings  to  the  table,  each 
simulation  took  hours  to  complete. 

The  new  methodology  has  led  to  a  new  way  of 
thinking  about  aircraft  response  to  in-flight  damage. 
By  moving  the  simulated  damage  from  one  place  to 


"The  new  methodology 
has  led  to  a  new  way 
of  thinking  about 
aircraft  response  to  in¬ 
flight  damage.  By 
moving  the  simulated 
damage  from  one 
place  to  another, 
certain  "vulnerable 
spots”  were  identified 
which  were  not 
previously 
anticipated.” 
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Figure  1.  Fighter  aircraft 
model  before  damage 


another,  certain  “vulnerable  spots”  were  identified 
which  were  previously  not  anticipated.  For  instance, 
at  some  flight  conditions  and  store  loading,  damage 
near  the  wingtip  was  seen  to  be  more  catastrophic 
than  the  same  damage  near  the  wing  root. 


New  Methods  Improve  Predicted 
Structural  Response 

This  research  is  significant  as  it  points  the  way  to  an 
effective  and  efficient  method  for  predicting  aircraft 
response  resulting  from  the  damage  from  detonation 
or  impact  of  a  missile  or  an  AAA  threat.  These 
results,  in  turn,  will  help  provide  a  pathway  for  test 
engineers  to  determine  the  most  appropriate  test 
method.  That  method  could  either  be  static, 
dynamic,  or  a  combination  of  the  two.  Subsequent 
concept  designs  of  dynamic  ground  tests  may  be 
based  on  improved  predicted  structural  response. 
This  work  will  be  used  in  the  future  to  assist  live  fire 
test  planners  in  designing  and  controlling  dynamic 
ground  test  to  more  realistically  reproduce  what 
actually  happens  in  flight. 
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During  the  engineering,  manufacturing,  and 
development  phase  of  the  F/A-18E/F 
program,  it  was  discovered  that  the 
F/A-18E  (Figure  1)  was  susceptible  to 
abrupt  wing  stall  (AWS).  Abrupt  wing  stall  occurs 
when  an  aircraft  undergoes  a  rapid,  severe  upper 
surface  flow  separation.  If  this  occurs 
asymmetrically,  large  rolling  moments  can  be 
produced,  which  result  in  uncommanded  lateral 
motion.  To  eliminate  AWS  on  the  F/A-1 8E,  the  flap 
schedule  was  modified  and  a  porous  door  was 
added.  However,  the  cause  of  the  F/A-1 8E’s  initial 
encounter  with  AWS  is  not  well  understood.  As  a 
result,  the  AWS  program  was  formed  to  investigate 
the  causes  of  AWS  and  to  develop  a  set  of  design 
guidelines  to  help  designers  avoid  AWS  on  future 
aircraft. 


Figure  1 .  The  F/A-1 8E  Super  Flornet,  the  Navy's  strike-fighter 
aircraft  (DoD  photo  by  Vernon  Pugh) 


Abrupt  wing  stall  has  been  traditionally  addressed 
through  expensive  flight  testing,  relatively  late  in  an 
aircraft  development  program.  With  advances  in 


CFD,  AWS  can  now  be  approached  using  CFD  to 
solve  the  RANS  equations.  However,  solving  these 
equations  for  this  class  of  flow  requires  large  memory 
resources  and  long  computational  times. 

During  the  redesign  of  the  F/A-1 8E  from  the  F/A-1 8C, 
the  wing  camber,  thickness,  twist,  leading-edge 
radius  and  leading-edge  flap-chord  ratio  were 
modified.  In  addition,  a 
leading-edge  snag  was 
added  to  the  wing.  Since 
the  F/A-1 8C  does  not 
experience  AWS,  these 
wing  parameters  are  likely 
contributing  to  the  AWS 
experienced  by  the  F/A 
18E.  Using  CFD,  these 
wing  parameters  were 
studied  independently  and  simultaneously  to 
determine  their  effect  on  AWS  and  to  develop  a  set  of 
design  guidelines. 

The  effects  of  these  six  wing  parameters  on  AWS 
were  analyzed  using  eleven  different  configurations. 
These  eleven  configurations  included  configurations 
in  which  one  or  more  of  the  wing  parameters  were 
modified.  For  example,  to  determine  the  effect  of 
twist  on  AWS,  the  twist  was  removed  from  the  F/A- 
1 8C  wing  to  form  one  of  the  configurations.  In  some 
cases,  two  wing  parameters  were  modified 
simultaneously.  For  example,  one  of  the 
configurations  was  obtained  by  adding  a  snag  to  the 
F/A-1 8C  wing  while  removing  the  camber. 

The  eleven  configurations  were  analyzed  at  two 
transonic  Mach  numbers  and  six  different  angles  of 
attack  (AoA).  This  resulted  in  132  different  cases, 
which  were  analyzed.  As  each  case  required 
approximately  6,000  hours  of  CPU  time  on  an  SGI 


“Current  results  indicate 
that  the  addition  of  the 
snag  and  the  reduction  of 
the  leading-edge  flap- 
chord  ratio  could  be 
contributing  significantly 
to  AWS  on  the  F/A-18E.” 
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Angle  of  Attack 

Figure  2.  Wing-root  bending-moment  coefficient  as  a  function  of  AoA  for  the  F/A-1 8C,  F/A-1 8C  with  a  snag,  F/A-1 8C  with  a  snag  and  reduced 
leading-edge  flap-chord  ratio  and  F/A-1 8E 


Origin,  extensive  CPU  time  and  computer  resources 
were  necessary  to  complete  this  task.  Through  using 
the  resources  of  the  HPCMP,  it  was  possible  to 
analyze  all  132  configurations. 

Although  this  research  is  ongoing,  meaningful  results 
have  been  obtained.  The  current  results  indicate  that 
the  addition  of  the  snag  and  the  reduction  of  the 
leading-edge  flap-chord  ratio  could  be  contributing 
significantly  to  AWS  on  the  F/A-1 8E.  In  Figure  2,  the 
wing-root  bending-moment  coefficient  is  plotted  as  a 
function  of  AoA  for  the  F/A-1 8C,  F/A-1 8C  with  a  snag, 
F/A-1 8C  with  a  snag  and  reduced  leading-edge  flap- 
chord  ratio  and  the  F/A-1 8E.  AWS  can  occur  when 
the  slope  of  this  curve  changes  sign.  It  can  be  seen 
from  the  figure  that  the  addition  of  the  snag  and 
reduction  of  the  leading-edge  flap-chord  ratio  on  the 
F/A-1 8C  moves  the  wing-root  bending-moment  curve 
toward  that  of  the  F/A-1 8E. 


The  results  of  this  research  also  indicate  that  the 
camber  may  be  contributing  to  AWS  on  the  F/A-1 8E, 
while  the  twist,  thickness  and  leading-edge  radius  do 
not  appear  to  be  impacting  AWS.  Although  this  is 
on-going  research,  the  team  expects  to  finish  the 
remaining  cases  expeditiously  so  that  they  can 
confirm  their  current  observations  and  contribute  to 
the  elimination  of  AWS  on  the  F/A-1 8E  and  other 
aircraft. 

CTA:  Computational  Fluid  Dynamics 

Computer  Resources:  SGI  Origin  2000/3000/3800 
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Currently,  fighter  development  is  under 
close  scrutiny  due  to  ever  increasing 
acquisition  costs.  The  Joint  Strike 
Fighter  program  made  low  cost  a  primary 
requirement  in  their  acquisition  strategy. 

Unfortunately,  there  is  little  confidence  that  CFD 
modeling  for  full  aircraft  is  accurate  under  massive 
separation,  limiting  its  usefulness. 

Reynolds  Averaged  Navier-Stokes 
Deficiencies 

While  advances  have  taken  place  in  areas  such  as 
grid  generation  and  fast  algorithms  for  solution  of 
systems  of  equations,  CFD  has  remained  limited  as  a 
reliable  tool  for  predicting  inherently  unsteady  flows 
at  flight  Reynolds  numbers.  Current  engineering 
approaches  to  predicting  unsteady  flows  are  based 
on  solution  of  the  RANS  equations.  The  turbulence 
models  used  in  RANS  methods  model  the  entire 
spectrum  of  turbulent  motions.  While  often  adequate 
in  steady  flows  with  no  regions  of  reversed  flow,  or 
possibly  exhibiting  shallow  separations,  it  appears 
that  RANS  turbulence  models  are  unable  to 
accurately  predict  phenomena  dominating  flows 
characterized  by  massive  separations.  Unsteady, 
massively  separated  flows  are  characterized  by 
geometry-dependent  and  three-dimensional  turbulent 
eddies.  These  eddies,  arguably,  are  what  defeat 
RANS  turbulence  models,  of  any  complexity. 


To  overcome  the  deficiencies  of  RANS  models  for 
predicting  massively  separated  flows, 

Detached-Eddy  Simulation  (DES)  was  proposed  with 
the  objective  of  developing 
a  numerically  feasible  and 
accurate  approach 
combining  the  most 
favorable  elements  of  RANS 
models  and  LES.  The 
primary  advantage  of  DES 
is  that  it  can  be  applied  at 
high  Reynolds  numbers,  as 
can  Reynolds-averaged 
techniques,  but  also  resolves  geometry-dependent, 
unsteady  three-dimensional  turbulent  motions  as  in 
LES. 

Testing  Detached-Eddy  Simulation 

Two  geometrically  simple  cases  were  examined  to 
test  the  method  and  build  confidence  for  full  aircraft 
simulation  —  a  delta  wing  at  27°  angle-of-attack 
exhibiting  vortex  breakdown,  and  a  forebody  at  90° 
angle  of  attack.  Although  the  geometries  are  simple, 
the  resulting  flows  are  quite  complex.  In  fact, 
accurately  predicting  delta  wing  vortex  breakdown  at 
high  Reynolds  number  has  challenged  researchers 
for  decades.  These  applications  are  then  used  to 
build  confidence  in  the  full  aircraft  simulations  (F-16, 
F-15E)  that  have  been  performed.  The  computations 
were  performed  using  the  unstructured-grid  solver 


“DES  applied  today  using 
HPC  resources  is  making 
more  “routine’’  the  full- 
configuration  prediction 
of  complex  geometries 
and  at  flight  conditions.’’ 


-60- 


2002  Success  Stories 


Cobalt.  Cobalt  is  the  commercial  version  of  the 
Common  High  Performance  Computing  Software 
Support  Initiative  code,  Cobalt60.  Grids  used  in  the 
calculations  of  all  the  geometries  summarized  below 
are  unstructured,  comprised  of  a  combination  of 
prisms,  pyramids,  and  tetrahedra. 

The  forebody  modeled  is  the  flow  at  90°  angle-of- 
attack  around  a  rectangular  ogive,  with  the  main 
cross-section  a  round  corner  square  as  shown  in 
Figure  1 .  The  Reynolds  number  based  on  the  width 
of  the  body  is  2.1  x  106.  Grid  sizes  for  the 
computations  performed  to  date  have  ranged  from 
4.2  x  1 06  to  5.2  x  1 06  cells.  Visualization  of  the 
pressure  distribution  on  the  surface  and  the 
instantaneous  vorticity  at  eight  stations  along  the 
body  are  shown  in  Figure  1 .  Predictions  obtained 
using  both  DES  and  RANS  (using  the  Spalart- 
Allmaras  one-equation  model)  are  shown.  Evident  in 
the  surface  pressure  prediction  from  the  RANS 
computation  is  that  along  the  forebody,  the  separated 
structure  is  more  coherent  than  that  obtained  in  the 
DES,  with  a  low-pressure  footprint  evident  along 
most  of  the  forebody.  The  DES  result  is 
characterized  by  a  more  chaotic  and  three- 
dimensional  structure  over  most  of  the  body, 
resulting  in  a  reasonably  constant  pressure  region 
along  the  top  (aft)  surface  (c.f.,  Figure  2).  Time- 
averaged  pressure  distributions  were  measured  in 
experiments  at  eight  axial  stations,  corresponding  to 
the  eight  locations  at  which  vorticity  contours  are 
shown  in  Figure  1 .  Compared  in  Figure  2  is  the 
pressure  coefficient  at  11%  of  the  forebody  length. 
The  angle  0=  0°  is  in  the  symmetry  plane  on  the 
windward  side,  with  0=  180°  in  the  leeward  side 
symmetry  plane.  As  shown  in  the  figure,  the  strong 
coherent  vortices  predicted  in  the  RANS  solution  give 
rise  to  a  large  variation  in  pressure  on  the  leeward 


Figure  1 .  RANS  (left)  and  DES  (right)  predictions  of  the  flow  around  the 
rectangular-ogive  forebody.  The  freestream  flow  is  at  90°  angle-of-attack 
(from  top  to  bottom  in  the  visualizations).  Surface  colored  by  pressure  and 
vorticity  contours  at  eight  stations.  The  location  of  the  eight  stations  is  the 
same  at  which  experimental  measurements  of  the  pressure  distribution  were 
available. 


Figure  2:  Pressure  distribution  around  the  rectangular  ogive 
forebody.  Comparison  of  DES  and  RANS  to  experimental 
measurements  at  station  3  (c.f.,  Figure  1). 


side  that  differs  markedly  from  the  experimental 
measurements.  The  largest  differences  in  the  RANS 
predictions  occur  closest  to  the  tip  of  the  forebody, 
resulting  in  large  errors  in  moment  predictions.  The 
DES  prediction  of  the  pressure  coefficient,  on  the 
other  hand,  is  in  excellent  agreement  with  the 
measurements,  a  result  of  the  more  accurate 
resolution  of  the  unsteady  shedding  that  yields  a  flat 
pressure  profile  on  the  leeward  side. 

The  70°  delta  wing  at  27°  angle-of-attack  has  also 
been  calculated.  A  comprehensive  grid  refinement 
study  was  performed  on  a  half-span  model  with  the 
number  of  cells  ranging  from  1 .2x1 06  to  1 0.5x1 06. 

The  finest  grid  results  are  shown  in  Figure  3. 
Numerous  flow  features  are  clearly  resolved 
including  shear  layer  instabilities  prior  to  vortex 
breakdown,  reverse  flow  near  vortex  breakdown,  post 
breakdown  windings,  and  vortex  shedding  off  the 
blunt  trailing  edge.  Resolved  turbulent  kinetic  energy 
in  a  line  along  the  vortex  core  is  plotted  in  Figure  4. 
DES  possesses  the  same  characteristics  as  LES,  in 
that  a  wider  range  of  turbulent  length  scales  is 
resolved  as  the  grid  density  is  increased.  The  finest 
grid  matches  the  peak  turbulent  kinetic  energy 
measured  experimentally.  Other  parameters  such  as 
velocity  in  the  core  of  the  vortex  and  breakdown 
location  were  in  excellent  agreement  with 
experiments.  Accurately  predicting  turbulent  kinetic 
energy  levels  provides  increased  confidence  that 
DES  will  be  able  to  predict  the  unsteady  loads  on  an 
aircraft  due  to  vortex  breakdown  (e.g.,  the  F-18  with 
tail  buffet). 
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Figure  3.  Flow  over  the  delta  wing  at  27°  angle  of  attack.  Isosurface 
of  vorticity. 


Figure  4.  Resolved  turbulent  kinetic  energy  in  the  core  of  the  vortex 
on  the  delta  wing 


Key  Advances  in  DES 

Work  to  date  has  provided  key  advances  in 
application,  assessment,  and  improvement  of  DES. 
Computation  of  building  block  flows  such  as  two- 
and  three-dimensional  forebodies  and  delta  wings 
has  established  a  foundation  for  resolution  of  several 
issues  important  to  using  DES  to  predict  the  flow  field 
around  full  aircraft,  helping  to  provide  guidance  in 
grid-generation  aspects,  turbulence  treatments,  and 
numerical  parameters.  The  subsequent  application 
of  DES  to  the  F-16  and  F-15E  has  further  enhanced 
the  confidence  level  with  lift  and  drag  predictions  that 
are  accurate  compared  to  flight  test  data.  These 
developments  will  soon  provide  aircraft  designers 
with  a  powerful  tool  for  the  prediction  of  massively 
separated  flows  over  complex  configurations  and  at 
flight  conditions.  The  availability  of  high  performance 
computing  continues  to  accelerate  the  development 
of  this  powerful  technique.  DES  applied  today  using 
HPC  resources  is  making  more  “routine”  the  full- 
configuration  prediction  of  complex  geometries  and 
at  flight  conditions.  Accuracy  in  lift,  drag,  and 
moment  predictions  around  the  F-15E,  for  example, 
are  within  5%  of  flight  test  data.  While  already 
relatively  accurate,  increases  in  HPC  capacity  will 
enable  efficient  simulation  using  substantially  larger 
grids.  DES  flow  fields  will  offer  richer  detail  on 
denser  grids,  with  associated  increases  anticipated  in 
overall  accuracy.  Increases  in  HPC  capacity  coupled 
with  the  emergence  of  more  efficient  and  accurate 
algorithms  will  soon  enable  prediction  of  additional 
physical  effects  such  as  fluid-structure  interactions. 
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Modern  and  future  aircraft,  such  as  the 
F-117  (Figure  1),  B-2,  F-22,  and  Joint 
Strike  Fighter  (JSF),  all  require  low 
observability  (LO)  to  gain  air  superiority 
and  increase  survivability.  This  significantly  increases 
the  development  costs  of  advanced  aircraft. 
Fortunately,  the  use  of  FIPC  enabled  Computational 
Electromagnetics  (CEM)  tools  is  reducing  LO  vehicle 
design  costs. 


Figure  1 .  US  Air  Force  ground  crew  members  do  a  final  inspection  of 
an  F-1 17A  Nighthawk  aircraft  before  it  departs  for  Kuwait  from 
Flolloman  Air  Force  Base,  NM.  (DoD  photo  by  Senior  Airman  Roberta 
McDonald,  US  Air  Force) 

A  team  of  scientists  and  engineers  led  by 
Dr.  Kueichien  Hill  of  AFRL,  other  DoD  agencies,  and 
Drs.  Navele  and  Sancer  of  Northrop  Grumman 
Integrated  Systems  are  developing  new  methods  to 
compute  accurate  radar  signatures  for  a  large  class 
of  inlet  and  exhaust  ducts  with  LO  features.  Engine 
inlet  and  exhaust  ducts  are  contributors  to  the  overall 
air  vehicle  signature.  In  addition  to  real  world 
geometries  for  explicit  aircraft,  ducts  with  generic 
material  treatments  and  shapes  were  used  to 
generate  a  radar  signature  database. 


Lowering  the  Costs  of  Modeling  and 
Testing 

Until  very  recently,  the  designs  of  low  radar  cross- 
section  vehicles  relied  almost  exclusively  on  costly 
experiments  with  some  support  of  computational 
electromagnetics.  While  first-principle  computer 
codes  satisfy  all  LO  requirements,  they  involve  the 
solution  of  large  linear 
systems  of  equations,  the 
order  of  which  increases  at 
the  cubic  power  of  the  radar 
frequency  being  studied  and 
with  the  complexity  of  the 
modeled  target.  The 
increasing  complexity  of 
modern  LO  weapon  systems 
coupled  with  higher 
frequencies  of  interest  has 
meant  that  such  simulations 
are  too  computationally 
intensive  for  even  the  most 
advanced  desktop  or  serial 
computers.  Massively 
parallel  supercomputers 
such  as  those  provided  by  the  HPCMP  must  be  used 
to  accurately  predict  the  radar  cross  section  of  real 
targets  at  higher  frequencies.  With  the  use  of  HPC 
resources,  several  large  test  models  (many  of  which 
cost  about  10  million  dollars)  could  be  eliminated. 
Not  only  could  model  cost  be  reduced,  but  the 
associated  test  cost  (on  the  order  of  $15,000  per 
measurement  shift)  could  also  be  reduced. 


"Massively  parallel 
supercomputers  such  as 
those  provided  by  the 
HPCMP  must  be  used  to 
accurately  predict  the 
radar  cross  section  of 
real  targets  at  higher 
frequencies.  With  the 
use  of  HPC  resources, 
several  large  test 
models  (many  of  which 
cost  about  10  million 
dollars)  could  be 
eliminated.” 
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CEM  Tools  Considerations 

In  order  to  be  useful  for  LO  designs,  CEM  tools  have 
to  satisfy  three  LO  imposed  CEM  requirements: 
accuracy,  material  modeling,  and  scalability.  Many 
CEM  codes  do  not  satisfy  all  three  LO  requirements. 
CEM  codes,  based  on  approximate  asymptotic  high 


frequency  techniques  such  as  the  XPATCH  code 
(developed  by  the  Automatic  Target  Recognition 
[ATR]  community)  cannot  satisfy  the  LO  accuracy 
and  material  modeling  requirements.  The  presence 
of  LO  material  treatments  and  the  close  coupling  of 
scattering  features  violate  the  fundamental 
assumptions  of  the  asymptotic  techniques.  CEM 
codes  based  on  the  finite  volume/difference  time 
domain  techniques  also  cannot  satisfy  the  LO 
accuracy  requirement.  They  employ  an  approximate 
radiation  boundary  condition,  which  causes  a  non¬ 
physical  reflection  that  contaminates  the  solution. 


Developing  a  Novel  Approach 

To  satisfy  all  three  LO  requirements,  Northrop 
Grumman  developed  a  first  principle-based  (solving 
Maxwell’s  equations  exactly  without  making  limiting 
assumptions)  hybrid  finite  element/method  of 
moments  (FEM/MoM)  code.  The  finite  element 
portion  of  the  code  allows  flexibility  in  material 
modeling  and  the  moment  method  portion  of  the 
code  generates  accurate  solutions  using  an  exact 
radiation  boundary  condition.  Under  the  support  of 
the  HPCMP’s  CHSSI  CEA-3  project  (Maturation  of  a 
LO  Component  Design  Code),  this  code  evolved  into 
a  robust,  portable,  and  scalable  code  that  was  used 
to  generate  the  radar  signature  database  for  engine 
inlet  and  exhaust  ducts. 

A  new  modular  duct  cascading  technique  was 
employed  to  efficiently  generate  the  radar  signature 


database,  especially  for  higher  frequency  bands. 

This  technique  allowed  the  doubling  of  the  duct 
length  without  repeated  computations.  This  level  of 
problem  size  and  accuracy  had  never  been 
demonstrated  before  and  was  only  possible  because 
of  HPC  resources.  Simulations  were  performed  with 
and  without  the  newly  discovered  technique  to 
corroborate  the  accuracy  of  the  new  technique  and 
to  demonstrate  the  several  orders  of  magnitude 
savings  in  run  time  and 
memory  requirement. 

Influencing  the 
Design  of  Engine 
Ducts 

Dr.  Hill’s  team’s  work  has 
influenced  the  design  of 
engine  ducts  in  three  different 
areas. 

First,  the  modular  duct 
cascading  technique  reduced 
the  simulation  time  by  several 
orders  of  magnitude  and 
produced  a  more  accurate 
solution  by  reducing  the  size  of  the  finite  element 
system.  Such  a  reduction  is  of  paramount 
importance  when  modeling  very  large  targets  as  the 
finite  element  technique  is  susceptible  to 
accumulation  of  numerical  errors.  Figures  1 , 2,  and  3 
illustrate  the  impact  of  the  modular  cascading 
technique  on  duct  design.  Figure  1  shows  the  radar 
cross  section  computed  using  the  cascading 
technique  for  a  duct  of  13  wavelengths.  The 
excellent  agreement  between  the  cascading 
technique  and  the  standard  FEM/MoM  technique  in 
Figure  1  is  lost  in  Figure  2,  which  illustrates  the  same 


"Radar  cross-section 
computations  for  the 
Defense  Advanced 
Research  Projects 
Agency's  (DARPA) 
future  strike  aircraft 
were  accomplished  in 
3.5  days  on  121 
processors  of  the 
Origin  3000  at  ARL. 
Such  a  computation 
would  have  been 
impossible  without  the 
use  of  HPCMP 
resources." 
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Figure  3.  Reduced  computational  requirement  of  innovative 
"cascading"  technique.  Computational  requirement  comparison  of 
standard  FEM-MoM  and  cascading  techniques. 


computations  but  for  a  longer  duct  of  25.8 
wavelengths.  This  disagreement  is  due  to  the 
numerical  errors  that  accumulate  in  the  standard 
FEM/MoM,  which  are  resolved  in  the  cascading 
technique  by  modeling  only  a  small  section  of  the 
duct,  performing  computations  on  the  small  section, 
and  cascading  such  computations  to  produce  the 
actual  target.  Such  a  cascading  procedure  yields 
greater  accuracy  and  saves  computation  time  as 
seen  in  Figure  3. 

The  second  major  impact  was  in  the  generation  of 
radar  images  for  duct  configurations.  Such  radar 
images  are  crucial  in  understanding  the 
phenomenology  of  radar  scattering  from  engine  and 


exhaust  ducts.  Figure  4  shows  typical  radar  images 
that  were  generated  on  the  ARL’s  Origin  3000. 

Thirdly,  the  work  influenced  the  simulation  of  future 
air  vehicles  including  the  engine  and  exhaust  ducts. 
Radar  cross-section  computations  for  the  Defense 
Advanced  Research  Projects  Agency’s  (DARPA) 
future  strike  aircraft  were  accomplished  in  3.5  days 
on  1 21  processors  of  the  Origin  3000  at  ARL.  Such  a 
computation  would  have  been  impossible  without  the 
use  of  HPCMP  resources. 
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CTA:  Computational  Electromagnetics  and  Acoustics 
Computer  Resources:  SGI  Origin  3000  [ARL]  and  IBM 
SP  P3  [ASC] 


Minimum  contour  =  40  dB 
Maximum  contour  =  20  dB 
Contour  step  =  5  dB 


Center  Frequency  =  0.2000  GHz 
Center  Angle  =  0.0  Deg 
Image  offset  =  (82.7,  D.D.) 

+/-  48  deg  Sector,  5  deg  step 


Duct  with  “pug”  indicated  in 
outline 


•  Low  downrange  and  resolution 
image  (left)  and  high  resolution 
(above) 


•  Generation  of  this  image  (no 
coating)  at  0.15  GHz  took  40 
hours  on  ARL  03K  (32  400  MHz 
process) 

•  Generation  of  a  SAR  image  for  a 
realistic  Duct  at  X-Band  will 
require  resources  of  an  order  of 
magnitude 


Figure  4.  Downrange  and  cross-range  image  of  generic  duct  at  very  high  frequencies  (center  -  0.15  GHz) 
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Scalable  Pre-  and  Post-Processing  Approaches  for 
Large  Scale  Computational  Electromagnetic 
Simulations 

Eric  Mark,  Jerry  Clarke,  Raju  Namburu,  and  Calvin  Le 
Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 
John  Neherbass 

Ohio  State  University  Columbus,  OH 


The  overall  objective  of  this  research  is  to 
develop  and  demonstrate  a  scalable 
computational  electromagnetic  software 
environment  to  address  accurate  prediction 
of  radar  cross  sections  (RCS)  for  full  range  armored 
vehicles  with  realistic  material  treatments  and 
complex  geometric  configurations.  A  software 
environment  consisting  of  scalable  pre-processing, 
post-processing,  and  accurate  finite-difference  time- 
domain  software  was  developed  which  demonstrated 
a  significant  reduction  in  overall  simulation  time  for 
large-scale  military  applications.  Computational 
electromagnetic  simulations  of  full  range  military 
vehicles  play  a  critical  role  in  enhancing  the  design 
and  performance  of  combat  systems  including  the 
Army’s  FCS.  This  high  fidelity  software  environment 
can  also  address  wide  band  communications 
applications  for  DoD. 

Developing  Scalable  Software  Tools 

Traditional  serial  pre-  and  post-processing 
approaches  cannot  generate  the  desired  meshes. 
They  are  also  inefficient  when  used  to  visualize  very 
large-scale,  high  fidelity  electromagnetic  simulations. 
To  overcome  this  problem,  a  team  of  scientists  at  the 
ARL  developed  a  suite  of  scalable  software  tools 
using  the  Interdisciplinary  Computing  Environment 
(ICE).  ICE  was  developed  at  ARL  for  generating 
desirable  meshes  from  the  available  computer  aided 
design  (CAD)  geometries  and/or  surface  models. 

For  computational  approaches  such  as  Method  of 
Moments  (MoM)  and  Finite  Difference  Time  Domain 
(FDTD)  codes,  simple  surface  model  information  is 
not  sufficient.  These  approaches  require  volumetric 
information  or  at  least  appropriate  thickness.  In 


addition  to  generating  desirable  meshes  on  scalable 
systems,  this  technique 
automatically  decomposes  the 
domain  for  efficient  load 
balancing.  In  addition,  these 
software  tools  provide  run  time 
visualization  of  very  large-scale 
data  from  multiple  processors. 

Applicability  of  the  approach 
was  demonstrated  for 
simulating  a  full-scale  ground 
vehicle  in  free  space  at  X-band 
using  scalable  finite  difference 
time  domain  software. 

Simulations  at  X-band  using  a 
finite  difference  time  domain  approach  require  2.56 


Figure  1 .  ZSU  23-4  combat  vehicle  subjected  to  an  electromagnetic  pulse  at 
X-band 


billion  cells  and  480  processors  of  IBM  SP3. 
Numerical  simulations  show  good  qualitative 
agreement  with  experimental  results.  Figure  1 
illustrates  the  capability  of  modeling  a  very  large- 
scale  application. 


"Computational 
electromagnetic 
simulations  of  full 
range  military  vehicles 
play  a  critical  role  in 
enhancing  the  design 
and  performance  of 
combat  systems 
including  the  Army’s 
Future  Combat 
Systems". 
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Software  tools  developed  under  this  research  along 
with  different  scalable  applications  software  create  a 
virtual  design  tool  for  addressing  large-scale  practical 
applications.  This  technology  will  assist  not  only  in 
optimizing  systems  performance  but  also  in 
performing  trade-off  studies  for  the  competing 
requirements  in  the  design  of  FCS. 
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Unsteady  RANS  Simulation  for  Surface  Ship 
Maneuvering  and  Seakeeping 

Joseph  Gorski 
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I  he  Chief  of  Naval  Operations  noted  in  his 
Top  Five  Priorities:  Future  Readiness: 

The  DD(X)  program  literally  outlines  what 
me  ship-borne  Navy  is  going  to  be  like  in 
the  next  30  years.  It’s  not  just  a  destroyer 
that’s  going  to  support  the  United  States 
Marine  Corps  with  a  precision  gun  that  is 
going  to  fire  a  round  from  75  to  1 00 
miles. . .  It  is  about  spiraling  that 
technology  off  into  the  missile  defense 
cruiser  of  the  future.  And  it  is  about 
creating  what  I  believe  is  one  of  our  most 
pressing  requirements,  a  littoral  combatant 
ship  that  is  going  to  operate  in  the  near¬ 
land  areas  and  address  real  threats  in  the 
future  in  near-land  anti-submarine  warfare 
and  in  mine  warfare  ...” 

As  today’s  Navy  continues  its  transformation,  the 
Naval  ships  of  the  future  will  be  fundamentally 
different  from  those  currently  in  the  fleet  in  order  to 
meet  evolving  missions  and  related  operational 
requirements  and  to  accommodate  emerging 
technologies  such  as  electric  drive  as  the  main 
propulsion  system.  Such  missions  include  increased 
littoral  operations  that  require  unprecedented  ship 
signature  reduction.  Because  these  ships  will  be 
radically  different  from  existing  designs,  there  is  no 
historical  database  to  benchmark  new  designs 
against.  Additionally,  seakeeping  and  maneuvering 
behavior  of  these  new  ships  is  significantly  different 
from  conventional  naval  ships.  The  US  Navy  needs  a 


way  to  analyze  these  new  hull  forms,  their  flow 
physics,  and  associated  operating  behavior  to 
properly  design  the  next  generation  of  naval 
combatants. 

Surface  ship  design  has  traditionally  been  developed 
by  the  build-and-test  approach.  Stealth  has  not  been 
part  of  the  major  performance  requirements. 
Maneuvering  and  seakeeping 
have  largely  been  addressed 
using  extensive  experiments. 

Historically,  simple  analysis 
methods  are  used  to  evaluate 
a  given  ship  under  a  large 
range  of  operating  conditions, 
but  these  existing  codes  rely 
heavily  on  experimental 
empiricism.  Consequently,  the 
simple  analysis  methods  often 
used  are  suspect  when  applied 
to  an  unconventional  shape 
without  an  existing  experimental  database.  This 
limits  the  applicability  of  such  techniques  for  future 
designs. 

Through  HPC  resources  it  is  becoming  possible  to 
predict  flow  phenomena  that  are  related  to 
hydrodynamic  seakeeping  and  maneuvering  analysis 
by  solving  the  RANS  equations  rather  than  by  simple 
strip  theory  approaches.  For  example,  the 
computation  of  Model  5415,  a  conventional  surface 
combatant,  with  rudders,  propeller  shafts,  support 
struts  and  rotating  propellers  at  436  rates  per  minute 


“HPC  resources 
enabled  the  team  to 
perform  the  first 
nonlinear  free  surface 
simulation  with  a 
surface  tracking 
approach  for  a  fully- 
appended 
conventional 
combatant  hull  form.” 
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(RPM),  required  over  5.7  million  nodes,  7.7  million 
prisms,  and  9.6  million  tetrahedra  for  its  unstructured 
mesh.  This  calculation  took  48  hours  on  75  IBM  SP3/ 
512  Mb  processors  (3600  processor  hours)  for  each 
propeller  revolution.  Using  HPC  environments,  the 
engineers  and  scientists  can  model  more  realistic 
physics  as  part  of  the  computation,  as  shown  in 
Figure  1.  The  Navy  can  now  use  a  computational- 
based  approach  to  address  new  designs.  It  is  only 
with  the  large  parallel  HPC  resources  available  in  the 
last  several  years  that  such  computations  could  be 
done  with  sufficient  accuracy  that  a  computational- 
based  approach  is  feasible. 

Enabling  the  First  Nonlinear  Free 
Surface  Simulation  for  a  Fully- 
Appended  Combatant  Hull  Form 

HPC  resources  enabled  the  team  to  perform  the  first 
nonlinear  free  surface  simulation  with  a  surface 
tracking  approach  for  a  fully-appended  conventional 
combatant  hull  form.  The  hull  form  was  similar  to 
DDG-51,  with  rudders,  propeller  shafts,  support 
struts,  and  rotating  propellers.  The  simulation  was 
performed  at  model-scale  for  steady  ahead  motion  in 
calm  water.  Such  calculations  allow  the  whole  flow 
field  to  be  studied,  unlike  traditional  experimental 
data,  which  supplies  limited  amounts  of  information. 
This  can  lead  to  better  understanding  of  the  flow 
physics  affecting  maneuvering  and  seakeeping. 
Additionally,  because  simpler  empirical-based 
methods  will  continue  to  be  needed  for  a 
considerable  time,  the  community  is  evaluating  using 
the  more  complicated  physics-based  calculations, 
where  new  hull  forms  can  be  analyzed  for  a  limited 
number  of  highly  complex  computations.  This  can 
provide  the  necessary  empirical  database,  such  as 
for  roll  damping  models,  for  the  simpler  codes.  This, 
in  turn,  will  reduce  the  need  for  building  and  testing 
new  models. 

The  Navy  team  has  had  promising  successes  using 
the  physics  based  calculations  in  their  research. 
Computations  have  been  done  for  increasingly 
complex  bodies.  Bilge-keel  forces  on  a  three- 
dimensional  rolling  cylinder  were  accurately 
predicted.  Detailed  flow  and  force  characteristics 
were  calculated  for  an  unappended  naval  combatant 
hull  in  prescribed  roll  motions.  Initial  calculations  for 
a  fully-appended  combatant  hull  gave  good 
qualitative  predictions  for  surface  pressure  and  free- 
surface  elevation  (Figure  2).  Calculations  for  a 
rudder-induced  turn  led  to  evaluation  of  improved 
methods  for  representing  a  free  surface. 


Figure  1 .  Wave  elevation  and  surface  pressure  distribution  on  Naval 
surface  combatant  Model  5415 


Future  Challenges  to  Increase  Insight 

Future  work  in  this  area,  while  challenging,  will  vastly 
expand  the  Navy’s  insight  into  the  transformational 
technologies  Admiral  Clark  envisioned.  A  surface¬ 
capturing  technique  appears  very  promising  in  terms 
of  robustness.  A  rudder-induced  maneuver  with  the 
surface-capturing  method  will  be  computed  for  the 
fully  appended  Model  5415.  Integration  of  the 
computed  viscous  stresses  and  pressure  distribution 
on  this  configuration  will  provide  the  hydrodynamic 
forces  and  moments  acting  on  the  ship.  Integration 
of  the  6-Degrees  of  Freedom  (DOF)  equations  using 
these  forces  and  moments  will  yield  the  time 
evolution  of  the  ship’s  velocity  and  rotation  rate.  The 
sensitivity  of  the  computed  trajectory  with  respect  to 
the  free  surface  will  also  be  examined  by  comparing 
this  maneuvering  solution  to  the  one  obtained  at 
Fr=0.  Future  efforts  will  also  include  unsteady 
simulations  of  Model  5415  in  incident  waves  with 
both  forced  and  free  roll  motions.  Initial  free  roll 
simulations  will  include  prediction  of  roll  decay  after 


Figure  2.  Time  sequence  of  axial  vorticity  contours  for  surface 
combatant  simulations  with  prescribed  roll  motion  using  CFDSHIR- 
IOWA 
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the  model  is  released  from  some  initial  angular 
displacement  at  f=0  and  will  allow  for  comparison 
with  experimental  measurements.  Planned 
simulations  for  Model  5415  with  forced  and  free  roll 
motions  will  be  considerably  more  complex  and 
computationally  demanding.  The  most  difficult 
challenge  to  address  is  computing  the  complexity  of 
the  air/water  interface  that  is  necessary  for  accurately 
predicting  ship  motions.  This  will  require  the  ability 
to  handle  breaking  waves  and  other  complexities 
such  as  water  on  deck. 
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Computational  Structural  Mechanics  Provides  a 
New  Way  to  "Test”  Behind-Armor  Debris 

Anand  Prakash 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 


The  Survivability/Lethality  Analysis  Directorate 
of  the  ARL  notes,  “In  survivability  analysis, 
the  ability  to  prevent  or  mitigate  personnel 
casualties  and  component  or  system 
destruction  is  directly  related  to  the  ability  to 
understand  and  analyze  the  process  of  how  damage 
occurs. 


Behind-armor  debris  (BAD)  effects  are  the  result  of  a 
combination  of  stochastic  events:  fragment  masses, 
shapes,  and  velocities,  as  well  as  the  shape  and 
dispersion  of  the  debris  cone  depend  upon: 


(1)  armor  composition 

(2)  munition  type 

(3)  attack  angle  (obliquity) 


Significant  reductions  in  debris  amount  and  effects 
will  increase  crew  survivability  chances  and  preserve 
the  system  and  its  subcomponents.  By  doing  this, 
we  better  the  chance  of  system 

_  survivability  and  afford  it  the 

opportunity  to  complete  its 
intended  mission.”1 


"This  is  an  important 
new  result,  as  there  is 
at  present  no  practical 
way  to  obtain  the 
velocity  distribution  of 
fragments  by  direct 
experimental 
measurement.” 


BAD  plays  a  critical  role  in  the 
evaluation  of  both  the 
survivability  of  crew  and 
components  in  combat  vehicles 
and  of  the  lethality  of  such 
weapon  systems.  The  US  ARL 
generates  BAD  data  at  its 
experimental  research  facilities  to  determine  the 
debris  characteristics  for  input  into  survivability 
analysis  codes.  Thus,  BAD  data  is  a  critical  input  to 
an  analysis  that  plays  a  central  role  in  every  major 
Army  acquisition  program,  including  the  FCS,  the 
Interim  Armored  Vehicle,  and  the  “Stryker”  (Figure  1), 
a  highly  deployable-wheeled  armored  vehicle  which 
will  be  the  combat  vehicle  of  choice  for  the  Army’s 
Interim  Brigade  Combat  Teams  (IBCTs). 


1  http://www.arl.army.mil/slad/Services/BAD.html 


Figure  1 .  A  Stryker  Infantry  Carrier  Vehicle  rolls  of  a  C-130  at  Fort  Irwin, 
CA,  after  being  transported  there  for  a  National  Training  Center  exercise 
(Photo  from  US  Army) 


Using  Physics  Based  Modeling  to 
Collect  BAD  Data 

Supplementing  this  experimental  capability  for  armor 
testing  is  an  important  goal.  Dr.  Anand  Prakash  of 
the  US  Army  Research  Laboratory  is  developing  a 
new  capability  to  obtain  BAD  characteristics  by  using 
physics-based  modeling  to  conduct  3-D  computer 
simulations.  The  expected  advantage  of  this 
approach  is  three  fold.  First,  it  will  reduce  costs  by 
reducing  the  number  of  experiments.  Second,  it  will 
provide  a  means  to  conduct  survivability  analyses 
involving  new  armors  before  they  are  actually 
manufactured.  Third,  it  is  capable  of  providing 
physical  details  of  BAD  (e.g.,  velocity  vectors  of 
fragments  and  the  ability  to  distinguish  the 
trajectories  of  penetrator  debris  from  armor  debris) 
that  are  difficult  to  obtain  experimentally.  Three- 
dimensional  simulations  of  BAD  were  conducted  with 
the  CTH  code  on  the  supercomputers  at  the  ARL 
MSRC.  CTH  is  an  Eulerian  wave  code  that  solves  the 
equations  of  continuum  mechanics  over  a  spatial 
mesh  in  small  time  steps.  The  material  properties  are 
provided  as  an  input.  Dr.  Prakash  used  the  Johnson- 
Cooke  model  of  plasticity  and  obtained  BAD  patterns 
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for  Kinetic  Energy  (KE)  long  rods  at  normal  impact  on 
rolled  homogenous  armor  (RHA)  of  finite  thickness. 
Wave  codes  like  CTH  are  suitable  to  obtain  the  net 
result  of  complex  wave  propagation,  reflection,  and 
interference  phenomena.  However,  the  high 
resolution  necessary  to  keep  track  of  behind  armor 
debris  in  three  dimensions  required  a  mesh  with  6 
million  cells  and  96  nodes,  with  200  CPU  hours  per 
node  for  complete  simulation.  Without  the  modern 
HPC  resources  of  the  MSRC,  it  would  not  have  been 
feasible  to  conduct  such  a  simulation.  These 
simulations  have  provided  new  information  on  the 
spatial  and  velocity  distribution  of  BAD  fragments. 

Capturing  the  Results 

Figure  2  shows  the  cross-sectional  view  of  an  early 
stage  of  a  3-D  simulation  in  which  a  rod  of  12.45  cm 
in  length  and  a  diameter  of  0.4  cm  strikes  a  steel 
armor  plate  1 1 .45  cm  thick  with  a  velocity  of  1 .6  km/s 
normal  to  the  plate.  A  thin  witness  plate  is  placed 
1 1 .45  cm  behind  the  armor  plate.  After  the  rod 
perforates  the  armor  plate,  the  behind  armor  debris 
strikes  the  witness  plate.  Figure  3  shows  the  BAD 
distribution  on  the  witness  plate  obtained  by 
simulation.  The  spatial  and  time  distributions  of 
various  physical  quantities  associated  with  BAD  were 
also  obtained.  For  example,  Figure  4  shows  the 
distribution  of  fragment  velocities  as  a  function  of 
radial  distance  from  the  point  of  intersection  of  the 
line  of  impact  with  the  back  surface  of  the  target 


BEHIND  ARMOR  DEBRIS.  3-D  SIMULATION  (*.  Prokosh) 


Figure  3.  Pattern  of 
behind  armor  debris 
captured  on  a  thin 
witness  plate  placed 
behind  the  target 
plate,  as  obtained  by 
a  3-D  simulation 
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Radial  Distance  (Proj.  Radii) 

Figure  4.  Fragment  velocity  (component  in  the  direction  of  initial  rod 
velocity)  as  a  function  of  radial  distance  of  origination  from  the 
impact  axis 


Figure  2.  The 
simulated  impact  of  a 
long  rod  on  a  thick 
armor  plate  to 
determine  the 
characteristics  of 
behind  armor  debris 
(BAD) 
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plate.  This  is  an  important  new  result,  as  there  is  at 
present  no  practical  way  to  obtain  the  velocity 
distribution  of  fragments  by  direct  experimental 
measurement.  Until  now,  survivability  analyses 
simply  made  assumptions  regarding  this  distribution. 


Impacting  the  Future 

The  team  is  now  performing  a  validation  study, 
comparing  the  results  of  this  simulation  approach  to 
actual  experimental  data.  When  this  is  completed, 
together  with  application  of  the  approach  to  further 
weapon/target  conditions,  the  Department  will  have  a 
powerful  and  flexible  new  capability  to  improve  the 
timeliness,  cost,  and  accuracy  of  vulnerability/lethality 
analysis  products  for  system  developers  and  the 
Army’s  independent  evaluators  and  decision  makers. 
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Airdrop  technology  is  a  vital  DoD  capability 
for  the  rapid  deployment  of  warfighters, 
ammunition,  equipment,  and  supplies.  In 
addition,  the  demand  for  airdrop  of  food, 
medical  supplies,  and  shelters  for  humanitarian  relief 
efforts  has  increased  significantly  in  recent  years. 
Airdrop  systems  development,  which  traditionally 
relies  heavily  on  time-consuming  and  costly  full-scale 
testing,  is  leaning  increasingly  on  modeling  and 
simulation.  Government  and  academic  researchers, 
who  have  teamed  to  make  computational  modeling  a 
valuable  tool  for  a  broad  range  of  airdrop 
applications,  are  addressing  a  variety  of  challenges. 
HPC  capabilities  are  being  used  to  numerically 
model  parachutes  and  airdrop  system  performance. 
These  capabilities  are  reducing  the  time  and  cost  of 
full-scale  testing,  minimizing  the  life  cycle  costs  of 
airdrop  systems,  assisting  in  the  optimization  of  new 
airdrop  capabilities,  and  providing  an  airdrop  virtual 
proving  ground  environment. 

Modeling  Fluid  Structure  Interaction 
Behaviors 

HPC  modeling  methods  are  being  developed  that 
can  be  applied  to  many  airdrop  systems  problems 
including  aerodynamics,  structural  dynamics,  and 
fluid-structure  interaction  (FSI)  behaviors.  These 


methods  are  based  on  the  Deforming-Spatial- 
Domain/Stabilized  Space-Time  (DSD/SST)  method 
for  the  fluid  and  a  semi-discrete  finite  element 
formulation  based  on  the 
principle  of  virtual  work  for  the 
cable-membrane  parachute 
structure.  An  iterative 
coupling  procedure  couples 
the  fluid  and  structure  along 
the  parachute  canopy 
surface.  Recent  applications 
simulations  have  focused  on 
the  aerodynamic  interaction 
between  two  separate 
parachutes  and  between 
different  canopies  in  a 
parachute  cluster,  FSI 
between  two  separate 
parachutes  and  four 
parachute  clusters,  FSI  for 
steerable  round  parachutes 
with  riser  control  inputs,  and 
the  FSI  experienced  by  parachute  soft-landing 
payload  retraction  systems.  In  addition  to  these 
application  simulations,  FSI  validation  simulations  are 
being  carried  out  and  numerical  results  are  being 
compared  with  water  tunnel  experiments  for  scale 
model  parachutes.  In  one  set  of  simulations,  the 
dynamics  of  a  small-scale  round  parachute  canopy 


“High  performance 
computing  capabilities 
are  being  used  to 
numerically  model 
parachutes  and  airdrop 
system  performance. 
These  capabilities  are 
reducing  the  time  and 
cost  of  full-scale  testing, 
minimizing  the  life  cycle 
costs  of  airdrop 
systems,  assisting  in  the 
optimization  of  new 
airdrop  capabilities,  and 
providing  an  airdrop 
virtual  proving  ground 
environment.” 
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are  compared  against  the  experimental  data  from  a 
similarly  scaled  canopy  for  validation  purposes.  The 
experiments,  which  were  carried  out  under  ARO 
sponsorship  and  using  AHPCRC  and  White  Sands 
Missile  Range  (WSMR)  resources,  have  sufficient 
spatial  and  temporal  resolution  to  permit  detailed 
comparison  with  the  simulation  results.  In  Figure  1 , 
the  three  snapshots  show  the  vorticity  field 
surrounding  two  parachutes  predicted  by  the  FSI 
software.  In  this  simulation,  the  lower  canopy  is  kept 
rigid  while  the  upper  parachute  deforms  due  to  its 
interaction  with  the  surrounding  flow  field. 
Deformation  of  the  upper  canopy  is  evident  as  it  is 
drawn  into  the  wake  of  the  lower  canopy  and 
approaches  canopy  collapse.  The  frames 
correspond  to  physical  times  of  0.00,  1 .75,  and  3.50 
seconds. 


Figure  1 .  The  three  snapshots 
show  vorticity  field  surrounding 
two  parachutes  predicted  by  the 
FSI  software.  In  this  simulation, 
the  lower  canopy  is  kept  rigid 
while  the  upper  parachute 
deforms  due  to  its  interaction  with 
the  surrounding  flow  field. 
Deformation  of  the  upper  canopy 
is  evident  as  it  is  drawn  into  the 
wake  of  the  lower  canopy  and 
approaches  canopy  collapse. 

The  frames  correspond  to 
physical  times  of  0.00,  1 .75,  and 
3.50  seconds. 


Figure  2.  This  figure  shows  the 
structural  response  of  the 
upper  canopy  as  it  deforms 
and  interacts  with  the 
surrounding  flow  field.  The 
colors  on  the  canopy 
correspond  to  the  differential 
pressures,  which  are  a  part  of 
the  fluid  dynamics  solution. 


challenges  associated  with  airdrop  systems 
modeling.  With  these  new  capabilities,  numerical 
simulations  are  being  carried  out  for  the  fluid 
dynamics,  structural  dynamics,  and  FSI  interactions 
experienced  by  airdrop  systems.  It  is  expected  that 
further  development  of  this  capability  will  result  in 
significant  reduction  of  life-cycle  costs  for  future 
airdrop  systems. 


CTA:  Computational  Fluid  Dynamics 

Computer  Resources:  Cray  T3E  [AHPCRC]  and  SGI 

Origin  2000  [WSMR] 


Figure  2  shows  the  structural  response  of  the  upper 
canopy  as  it  deforms  and  interacts  with  the 
surrounding  flow  field.  The  colors  on  the  canopy 
correspond  to  the  differential  pressures,  which  are  a 
part  of  the  fluid  dynamics  solution. 

Summary 

New  computational  technologies  based  on  state-of- 
the-art  methods  are  being  developed  to  advance 
airdrop  systems  modeling  capabilities.  These 
technologies  are  necessary  to  address  a  variety  of 
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3-D  Bomb  Effects  Simulations  for  Obstacle 
Clearance 

Alexandra  Landsberg 

Naval  Surface  Warfare  Center  (NSWC),  Indian  Head,  MD 


The  Navy  needs  a  system  capable  of 

simultaneously  breaching  obstacles  and 
clearing  mines  during  an  amphibious 
assault.  The  rapid  creation  of  transit  lanes 
through  shoreline  defenses  allows  landing  craft  to 
deposit  troops  and  equipment  directly  onto  and 
beyond  the  beaches.  The  Navy  and  Marine  Corps 
needs  the  capability  to  conduct  an  efficient 
amphibious  assault,  minimizing  the  number  of  sorties 
required  to  conduct  mine  and  obstacle  clearance, 
and  to  minimize  amphibious  lift  requirements.  In  the 
past  few  years,  PMS-490  and  the  ONR  have 
sponsored  brainstorming  sessions  and  Broad  Area 
Announcements  to  generate  concepts  for  obstacle 
clearance.  In  response  to  the  obstacle-breaching 
concern,  the  ONR  started  the  Bomb  Effects  for 
Obstacle  Clearance  Program,  along  with  several 
other  alternative  mine/obstacle-breaching  programs. 
The  Bomb  Effects  Program  is  a  potentially  effective 
means  of  defeating  obstacles  and  mines  in  the  surf 
zone  and  beach  zone. 

This  is  a  highly  visible,  high  priority  problem  that 
supports  the  Navy’s  investment  in  organic  mine 
countermeasures  (MCM),  and  is  part  of  the  Navy’s 
Future  Naval  Capabilities  (FNC)  investments  in 
organic  mine  countermeasures.  This  weapon  system 
is  part  of  the  Analysis  of  Alternatives  (AoA)  for  rapid, 
organic,  standoff  technologies  for  breaching  surf 
zone  and  beach  zone  mines  and  obstacles.  This 
effort  will  provide  important  input  into  that  analysis  for 
comparing  Bomb  Effects  to  other  alternatives. 

The  Bomb  Effects  Program 

The  Bomb  Effects  Program  is  based  on  using 
standard  general-purpose  bombs  as  an  existing, 
rapidly  deployable,  building  block  for  developing  an 
effective  system  against  obstacles.  Two  components 
in  this  program  are  a  testing  component  and  an 
analytic  component.  The  goal  of  the  testing 
component  is  to  study,  identify,  and  verify  the 


damage  mechanisms  of  obstacles,  both  on  land  and 
in  water,  subjected  to  multiple  bomb  detonations. 
However,  only  limited  experimental  and  full-scale 
testing  of  bomb  effects  has  been  and  can  be 
performed.  The  goal  of  the 
analytic  component  is  to 
generate  semi-analytic/ 
empirical  models  that, 
when  incorporated  in 
system  effectiveness 
studies,  will  determine  the 
lethality  of  general-purpose 
bombs  against  obstacles  in 
the  beach  zone  and  surf 
zone.  In  order  to  generate 
these  analytic  models, 
experimental  data  or 
hydrocode  data  is  required.  Prior  modeling  and 
simulation  techniques  generated  a  flow  field  due  to  a 
bomb  detonation  and  then  applied  the  loads  to 
determine  the  response  of  the  obstacle.  This 
decoupled  method  was  the  only  feasible  method 
since  the  applicable  hydrocodes  were  either  not  well 
advanced  or  were  not  parallelized  at  the  time. 

This  3-D  modeling  and  simulation  effort  complements 
both  the  experimental  and  analytic  efforts  in 
providing  data  that  would  otherwise  be  difficult  to 
obtain  experimentally.  These  “live-fire”  simulations 
are  effectively  an  operational  test  of  the  real-life 
scenario  of  multiple  bombs  detonated  in  the  beach 
and  surf  zone. 

Applying  High  Performance 
Computing  Resources  to  Live-Fire 
Simulations 

The  parallel  hydrocode  Alegra  is  being  used  for 
these  simulations.  In  Alegra,  a  problem  is  divided 
into  blocks;  each  block  can  then  be  run  as  Eulerian, 
Lagrangrian,  or  Arbitrary  Lagrangian  Eulerian  (ALE). 


"This  is  a  highly  visible, 
high  priority  problem  that 
supports  the  Navy’s 
investment  in  organic  mine 
countermeasures  (MCM), 
and  is  part  of  the  Navy’s 
Future  Naval  Capabilities 
(FNC)  investments  in 
organic  mine 
countermeasures.” 
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The  flexibility  to  blend  Eulerian,  Lagrangian,  and  ALE 
methods  is  appealing  for  modeling  these  types  of 
surf  zone  simulations  since  an  Eulerian  solver  can 
accurately  capture  the  shocks  in  water  and/or  air,  the 
Lagrangian  elements  most  accurately  model  the 
tetrahedons  and  the  Eluerian  or  ALE  solver  can  be 
used  to  model  the  sand.  The  complex  shock 
interactions  of  multiple  bombs  in  water  and  sand 
strongly  influence  the  loads  experienced  on  these 
obstacles.  The  goal  is  to  model  the  full-scale  tests  to 
determine  the  impact  of  bomb  detonations  on 
tetrahedron  obstacles  as  a  function  of  time. 
Simulations  include  modeling  subsets  of  the  full- 
scale  tests  as  well  as  the  full  domain. 

Due  to  advances  in  parallel  hydrocodes  and  the 
availability  of  HPC  resources,  this  effort  is  now  able  to 
model  scenarios  that  can  not  be  tested 
experimentally,  solved  analytically,  or  modeled  using 
earlier  computational  methods.  Specifically,  the 
ability  to  model  a  single  bomb  detonation  impacting 
multiple  obstacles  was  lacking.  This  project  has 
demonstrated  a  single  bomb  detonation  impacting 
multiple  objects  for  both  a  detonation  in  air  and  for  a 
detonation  in  water.  More  complex  scenarios 
including  multiple  bombs  detonated  simultaneously 
or  a  bomb  sequentially  impacting  multiple  obstacles 
can  now  be  performed. 

A  general  methodology  was  developed  for  mesh 
generation,  flow  solver  execution,  and  post¬ 
processing  on  DoD  HPC  platforms.  This 
methodology  included  linking  Patran®  to  multiple 
Sandia  National  Laboratory  software  packages  and 
then  linking  to  post-processing  software  including 
Ensight®  and  CTHPLT.  The  development  of  a 
general,  efficient  methodology  allows  for  rapid  model 
generation,  on  the  order  of  hours  rather  than  days. 

Simulations  were  typically  run  to  several  milliseconds 
(1-6  ms).  These  times  were  based  on  either  time 
history  data  from  full-scale  tests  or  the  computational 
domain  size.  Simulations  performed  included  (1)  Mk 
82  and  Mk  83  bombs  detonated  in  air  and  in  water  on 
a  sand  surface,  (2)  two  Mk  83  bombs  8  feet  apart 
detonated  simultaneously  in  air  on  a  sand  bottom,  (3) 
five  Mk  83  bombs  32  feet  apart  detonated 
simultaneously  in  air  on  a  sand  bottom  (typical 
separation  distance  for  air  detonations)  and  8  feet 
apart  (typical  separation  distance  for  underwater 
detonations),  (4)  Mk  83  bomb  detonated  in  air  near  a 
“leg”  of  tetrahedron  on  a  sand  bottom  (smaller 
physical  domain,  simulation  used  for  debugging  and 
analysis),  (5)  Mk  83  bomb  detonated  in  air  near  a 
tetrahedron  on  a  sand  bottom  at  4  foot  and  8  foot 
stand-off  (separation  distance  for  full-scale  tests  in 
air),  and  (6)  Mk  82  bomb  detonated  in  air  and  in 
water  near  4  tetrahedrons  on  a  sand  bottom  with  8 
foot,  1 2  foot,  1 6  foot,  and  20  foot  standoffs. 


The  following  is  a  highlight  resulting  from  a  Mk  83 
bomb  detonated  in  air  near  a  single  tetrahedron. 
Modeling  and  simulation  of  the  impact  of  a  bomb 
detonation  near  a  full  tetrahedron  is  difficult  due  to 
the  large  mesh  size  required  to  accurately  mesh  the 
geometry  of  the  tetrahedron  as  well  as  the  large 
physical  domain  surrounding  the  bomb  and 
tetrahedron.  Each  leg  of  the  tetrahedron  is  a  5  foot 
long  by  4  inches  wide  angle  iron  by  5/8  inch  thick 
steel.  A  tabular  Sesame  EOS  with  an  elastic-plastic 
model  was  used  to  model  the  steel  tetrahedron  leg. 
Figure  1 ,  in  which  the  tetrahedron  is  4  feet  from  the 
bomb,  shows  an  isocontour  of  the  high  explosives 
(HE)  gas  as  it  impinges  on  the  tetrahedron  at  t  =  275 
ms.  The  isocontour  and  a  centerline  slice  are 
colored  by  pressure.  The  pressure  on  the  forward 
leg  of  the  tetrahedron  is  increasing  as  well  as 
beginning  to  deform  the  tetrahedron. 


Figure  1 .  Mk  83  detonated  in  air  near  a  single  tetrahedron. 
Isocontour  of  HE  gases  colored  by  pressure. 


In  addition,  simulation  of  a  Mk  82  bomb  detonated  in 
air  and  in  water  near  four  tetrahedrons  was 
performed.  One  of  the  successes  of  this  effort  has 
been  the  modeling  and  simulation  of  a  bomb 
detonation  impacting  multiple  tetrahedrons  in  air  and 
in  water.  Figure  2  is  the  detonation  in  air  at  t= 1  ms. 
This  figure  shows  an  isocontour  of  HE  gas  colored  by 
velocity  magnitude.  These  simulations  would  not  be 
possible  without  HPC  resources. 


Figure  2.  Mk  82  detonated  in  air  near  four  tetrahedrons.  Isocontour 
of  HE  gases  colored  by  velocity  magnitude. 
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Future  Efforts 

To  date,  the  simulations  have  been  run  in  pure 
Eulerian  mode.  One  of  the  key  remaining  challenges 
is  to  run  these  simulations  with  Eulerian,  Lagrangian, 
and  ALE  blocks.  The  interfaces  between  these 


blocks  has  been  problematic.  These  issues  are 
being  addressed  both  in-house  and  through 
consulting  with  Sandia  National  Laboratories,  the 
developers  of  Alegra,  and  the  staff  of  the  ARL,  who 
also  use  the  Alegra  hydrocode. 


CTA:  Computational  Structural  Mechanics 
Computer  Resources:  SGI  Origin  3800  [ARL,  SMDC] 
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Providing  the  Warfighter  Information  Superiority 
in  Littoral  Waters 

Richard  Allard,  Alan  Wallcraft,  Cheryl  Ann  Blain,  Mark  Cobb,  Lucy  Smedstad,  Patrick 
Hogan,  and  Timothy  Keen 

Naval  Research  Laboratory  (NRL),  Stennis  Space  Center,  MS 
Stacy  Howington,  Charlie  Berger,  and  Jane  McKee  Smith 
Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 
Matt  Bettencourt 

Center  for  Higher  Learning/University  of  Southern  Mississippi,  Stennis  Space  Center,  MS 


The  littorals  of  the  world  are  areas  of  great 
strategic  importance.  Approximately  95% 
of  the  world’s  population  lives  within  600 
miles  of  sea,  80%  of  all  countries  border  the 
coast,  and  80%  of  the  world’s  capitals  lie  within  300 
nautical  miles  of  shore.  With  the  end  of  the  cold  war, 
DoD’s  focus  has  shifted  from  a  large  land/ 
sea  battle  scenario  with  a  monolithic 
global  adversary  to  dealing  with  low- 
intensity  conflicts  in  the  near-coastal  or 
littoral  regions.  In  wartime,  forces  that 
dominate  the  littoral,  including  the 
undersea  environment,  operate  with 
impunity  in  the  face  of  area  denial  threats 
while  taking  initial  action  to  defeat  those 
threats  and  prepare  the  battlespace  for 
follow-on  forces. 

Information  superiority — understanding 
and  exploitation  of  the  natural 
environment — is  critical  to  the  safe  and 
effective  operation  of  joint  forces  engaged 
in  littoral  expeditionary  warfare.  The 
tactical  advantage  will  depend  on  who 
can  most  fully  exploit  the  natural 
advantages  gained  by  a  thorough 
understanding  of  the  physical  environment, 
aspect  of  the  littoral  environment  is  critical  to 
conducting  military  operations,  a  huge  challenge  to 
both  the  warfighters  and  the  Meteorology  and 
Oceanography  (METOC)  community  that  supports 
them. 


Characterizing  the  Battlespace 
Environment 

Characterization  of  the  battlespace  environment  to 
ensure  optimal  employment  of  personnel,  systems 
and  platforms  depends  on  continuous  improvement 
in  science  and  technology.  Key  and 
essential  elements  in  that  characterization 
are  effective,  relocatable,  high-resolution, 
dynamically  linked  mesoscale, 
meteorological,  coastal  oceanographic, 
surface  water/groundwater,  riverine/ 
estuarine,  and  sediment  transport  models. 
The  linkages  between  these  models, 
coupled  with  the  complexity  and  the  scale 
of  the  environment,  require  the 
implementation  of  focused  sets  of  these 
models  on  DoD  HPC  architectures  in 
support  of  real  time  operations,  scenario 
(operational  and/or  training)  planning,  and 
course  of  action  analysis. 

The  High  Fidelity  Simulations  of  Littoral 
Environments  (HFSOLE)  Portfolio  is 
funded  by  CHSSI.  The  HFSOLE  effort 
encompasses  four  projects;  1)  the  System  Integration 
and  Simulation  Framework  Project  is  developing  the 
interface  algorithms  for  the  atmospheric,  riverine, 
estuarine,  wave,  tidal,  and  ocean  models;  2)  the 
Near-shore  and  Estuarine  Environments  Project  is 
coupling  and  enhancing  nearshore  process  models 
to  efficiently  simulate  interactions  between  winds, 
waves,  currents,  water  levels,  and  sediment 
transport;  3)  the  Surface  Water/Groundwater,  Riverine 


“The  HFSOLE  models  have 
been  shown  in  literature  to 
provide  a  realistic  depiction 
of  the  complex  physical 
processes  that  occur  in  the 
littoral  battlespace,  from 
rivers  and  estuaries  to 
oceanic  waters.  CHSSI  has 
given  DoD  an  opportunity  to 
make  these  large  numerical 
codes  scalable  on  a  host  of 
HPC  computational 
platforms  and  greatly 
reduce  the  wallclock  time 
needed  to  perform  a 
simulation,  hindcast,  or 
forecast.  ’’ 


Every 
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and  Tidal  Environments  Project  is  enhancing  the 
capabilities  of  the  Adaptive  Hydraulics  (ADH)  model. 
This  project  will  provide  DoD  the  tools  to  generate 
fast  and  accurate  estimates  of  water  depths  and 
velocity  distributions  for  use  in  ingress/egress 
assessments  for  rivers  and  streams.  They  will  also 
support  flow  rate  and  sediment  loading  as  input  for 
coastal  simulation,  and  ground-surface  inundation 
and  moisture  content  predictions  for  use  in  local 
atmospheric  simulations.  The  fourth  project,  the 
System  Applications  Project,  is  improving  the  fidelity 
and  execution  speed  of  DoD  maritime  operation 
simulation  systems  dependent  on  data  from  ocean 
surface  behavior  models.  Such  simulations  include 
amphibious  assault,  mine  clearing,  and  sea  keeping 
operations.  This  project  provides  for  the 
development,  implementation,  and  testing  of  an 
environmental  server  to  support  Modeling  and 
Simulation  (M&S). 

Modeling  the  Littoral  Waters 

Previously,  these  models  ran  independently,  without 
any  coupling  or  linkages,  and  typically  in  serial 
fashion  on  one  processor.  The  HFSOLE  Portfolio 
includes  eight  numerical  ocean,  wave,  and  riverine 
models.  They  use  HPC  resources  to  solve  for  a  host 
of  equations  including  the  shallow-water  equations, 
primitive  continuity  and  momentum  equations,  and 
the  spectral  action  balance  equation.  For  example, 
the  evolution  of  the  wave  spectrum  in  the  Simulating 
WAves  Nearshore  (SWAN)  (Booij  et  al.  1999)  wave 
model  is  described  by  the  spectral  action  balance 
equation,  which  for  Cartesian  coordinates  is: 

— N  +  — c„N  +  — c,.N  +  — +— Crt,N  =  —  (1) 

dt  5s  5y  }  rta  T  to  cr 

The  first  term  in  the  left-hand  side  of  this  equation 
represents  the  local  rate  of  change  of  action  density 
in  time.  The  second  and  third  terms  represent 
propagation  of  action  in  geographical  space  (with 
propagation  velocities  cx  and  cy  in  x-  and  y-space, 
respectively).  The  fourth  term  represents  shifting  of 
the  relative  frequency  due  to  variations  in  depths  and 
currents  (with  propagation  velocity  cjn  cr-space). 

The  fifth  term  represents  depth-induced  and  current- 
induced  refraction  (with  propagation  velocity  c0  in  0- 
space).  The  expressions  for  these  propagation 
speeds  are  taken  from  linear  wave  theory.  The  term 
S  (=  S(o,0))  at  the  right  hand  side  of  the  action 
balance  equation  is  the  source  term  in  terms  of 
energy  density  representing  the  effects  of  generation, 
dissipation,  and  nonlinear  wave-wave  interactions. 


The  HFSOLE  models  have  been  shown  in  literature 
to  provide  a  realistic  depiction  of  the  complex 
physical  processes  that  occur  in  the  littoral 
battlespace,  from  rivers  and  estuaries  to  oceanic 
waters.  CHSSI  has  given  DoD  an  opportunity  to  make 
these  large  numerical  codes  scalable  on  a  host  of 
HPC  computational  platforms  and  greatly  reduce  the 
wallclock  time  needed  to  perform  a  simulation, 
hindcast,  or  forecast.  Additionally,  the  Model 
Coupling  Environmental  Library  (MCEL)  being 
developed  in  HFSOLE  allows  for  the  coupling  of 
these  numerical  models  running  on  distributed  HPC 
platforms.  The  distributed  model-coupling 
infrastructure  uses  a  data  flow  approach  where 
coupling  information  is  stored  in  a  centralized  server 
and  flows  through  processing  routines  or  filters  to  the 
numerical  models.  The  resulting  software  is  expected 
to  provide  substantial  improvements  over  existing 
technologies  by  including  inter-model  interactions. 
The  models  are  used  by  a  DoD  Challenge  Project 
that  allows  for  more  experimentation  and  the  ability  to 
perform  simulations  at  higher  resolutions. 

Japan/East  Sea  Circulation 
Simulations 

Using  HPC  resources,  HFSOLE  researchers  are 
performing  complex  computations  in  a  timely 
manner.  This  will  greatly  impact  providing  the 
warfighter  with  a  tactical  advantage.  An  example  of 
this  time  savings  is  illustrated  with  the  coastal  ocean 
model  ADvanced  CIRCulation  Model  (ADCIRC)  that 
was  applied  to  the  simulation  of  rip  currents 
generated  by  breaking  waves.  As  shown  in  Figure  1 , 
barred  beaches  with  channels  often  produce  rip 
currents  (at  the  channels)  when  incident  waves  break 
in  the  vicinity  of  the  bars.  The  laboratory  wave-driven 
experiments  of  Haller  and  Dalrymple  (1999)  were 
simulated.  The  dimensions  of  the  beach  are  20  m  by 
1 8.2  m  with  a  water  depth  variation  from  0.37  m 


Cross-shore  Distance  (m) 


Figure  1 .  ADCIRC  model  computed  circulation  of  the  steady  state 
sea  surface  height  and  current  velocity  over  a  laboratory  barred 
beach.  Note  the  presence  of  rip  currents  in  the  two  channels. 
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offshore  to -0.01  m  at  the  shoreline.  The 
computational  domain  is  discretized  to  a  spacing  of 
0.1  m  using  40,040  elements  and  20,313  nodes.  The 
simulation  extends  0.6  days  to  a  steady  state  with 
computations  made  every  0.025  seconds,  producing 
a  rip  current  in  the  upper  channel.  Computational 
times  dropped  from  123  hours  using  one  CPU  to  2 
hours  using  128  CPUs. 

A  series  of  ocean  circulation  simulations  in  the 
Japan/East  Sea  have  been  performed  on  the  NAVO 
Cray  T3E.  The  model  used  is  a  Hybrid  Coordinate 
Ocean  Model  (HYCOM),  a  next  generation  circulation 
model  under  development  at  the  Naval  Research 
Laboratory  (NRL),  the  University  of  Miami,  and  Los 
Alamos  National  Laboratory.  Ideally,  an  ocean 
general  circulation  model  should  (a)  retain  its  water 
mass  characteristics  for  centuries  (a  characteristic  of 
isopycnal  coordinates),  (b)  have  high  vertical 
resolution  in  the  surface  mixed  layer  (a  characteristic 
of  z-level  coordinates),  and  (c)  have  high  vertical 
resolution  in  surface  and  bottom  boundary  layers  in 
coastal  regions  (a  characteristic  of  terrain-following 
coordinates). 

The  Japan/East  Sea  has  been  used  to  test  the 
robustness  of  HYCOM  because  it  contains  many 
circulation  features  found  in  the  global  ocean,  such 
as  a  western  boundary  current,  subpolar  gyre,  and 
rich  eddy  field,  and  it  is  small  enough  to  allow 
simulations  with  very  high  grid  resolution.  Under  this 
HPC  DoD  Challenge  Project,  several  simulations  with 
3.5  km  resolution  were  performed  for  several  years 
each.  Scalability  was  via  MPI/SHared  MEMory 
(SHMEM),  or  OpenMP,  or  both,  although  SHMEM 
was  used  exclusively  for  the  simulations  performed 
on  the  T3E.  The  mesh  size  for  the  high-resolution 
configuration  is  394x618x15,  and  a  one-year 
simulation  requires  about  12,000  CPU  hrs/model 


Figure  2.  Impact  of  progressively  increasing  the  horizontal  grid 
resolution  using  HYCOM.  At  14  km  resolution,  unrealistic  overshoot 
of  the  East  Korea  Warm  Current  (130°E,  39°N)  is  evident.  This 
overshoot  is  diminished  at  7.5  km,  and  at  3.5  km  the  current 
separates  from  the  coast  at  the  observed  latitude.  The  strength  of 
the  subpolar  gyre  strengthens  as  the  horizontal  resolution  is 
increased. 


year.  The  HYCOM  simulations  have  shown  (Figure  2) 
that  high  horizontal  grid  resolutions  are  needed  to 
adequately  resolve  the  mesoscale  features  as  well  as 
the  coastline  and  bottom  topography,  especially  over 
the  shelf. 

Software  Testing  to  Begin 

Ongoing  work  in  HFSOLE  includes  preparation  for 
individual  Alpha  and  Beta  software  tests  for  each  of 
the  models  included  in  the  portfolio.  A  portfolio  wide 
Alpha  test  using  the  MCEL  software  took  place  in 
January  2003.  Inputs  to  these  models  include 
topographically  controlled  mesoscale  atmospheric 
forcing  such  as  wind  stress,  sensible  and  latent  heat 
fluxes,  incoming  solar  radiation  and  sea  level 
pressure,  high-resolution  bathymetry  and  tidal 
constituents.  The  final  results  of  this  effort  will  be  a 
scalable  suite  of  dynamic  models  for  real-time 
operational  applications  and  high-fidelity  simulations 
for  modeling  and  simulation  scenarios. 

References 

Booij,  N.,  R.C.  Ris  and  L.H.  Holthuijsen,  “A  Third- 
Generation  Wave  Model  for  Coastal  Regions  1.  Model 
Description  and  Validation,”  J.  Geophys.  Res.,  Vol.  104, 

No.  C4,  pp.  7649-7666,  April  1999. 

Haller,  Merrick  C.,  and  Robert  A.  Dalrymple,  “Rip  Current 
Dynamics  and  Nearshore  Circulation”,  Res.  Report  CACR- 
99-05,  Center  for  Applied  Coastal  Research,  Univ.  of 
Delaware,  1999. 


CTA:  Climate/Weather/Ocean  Modeling  and  Simulation 
and  Environmental  Quality  Modeling  and  Simulation 
Computer  Resources:  Cray  T3E  [ERDC,  NAVO],  SGI 
Origin  3000  (ERDC],  and  IBM  SP,  Cray  SV1,  and  SUN 
El  0000  [NAVO] 


-80- 


2002  Success  Stories 


Getting  the  Waves  Right 

Jane  McKee  Smith 

US  Army  Corps  of  Engineers,  Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 


Ocean  waves  near  the  coast  have  a  huge 
impact  on  military  activities  aimed  at 
getting  soldiers,  sailors,  marines,  and 
equipment  on  the  shore.  Although  waves 
are  generated  in  the  open  ocean  and  propagate  over 
long  distances,  the  last  kilometer  or  less  near  the 
shore  is  the  critical  region  for  landing  and  docking. 
Nearshore  wave  transformation  is  solved  in  the 
model  STeady-state  WAVE  (STWAVE)  using  the  wave 
action  balance  equation.  The  model  includes  the 
processes  of  shoaling  and  refraction  (increases  in 
wave  height  and  changes  in  wave  direction  in 
shallow  water),  wave-current  interaction  (changes  in 
wave  properties  caused  by  tidal  or  nearshore 
currents),  wave  growth  due  to  the  wind,  and  wave 
breaking.  The  model  is  based  on  linear  wave  theory 
and  does  not  include  reflection  or  bottom  friction.  By 
linking  the  nearshore  wave  transformation  model 
STWAVE  with  deepwater  wave  forecasts  or  hindcasts 
(at  large  spatial  scales),  nearshore  waves  can  be 
estimated  on  beaches  and  at  harbor  and  inlet 
entrances.  Forecasts  are  used  for  short-term 
planning  of  operations,  and  hindcasts  are  used  to 
garner  climatological  information  for  longer-term 
planning  (typical  and  extreme  conditions,  seasonal 
variation,  storm  probability). 


efficient  parallelization  allows  improved  resolution  of 
the  nearshore  domain  (which  improves  model 
accuracy)  and  more  frequent  model  runs  (more 
resolution  in  time),  while  still  meeting  or  beating 
required  forecast  time  constraints.  Improved 
nearshore  modeling  times  also  allows  more  time  for 
other  modeling  components 
and  for  decision  making. 

Figure  1  shows  verification  of 
the  parallel  version  of 
STWAVE  at  Duck,  NC,  for  the 
year  1997.  Full-year 
simulations  were  not  run  prior 
to  the  parallel  capability. 

Continuing  challenges 
include  expanding  to  more 
HPC  platforms  and  a  larger 
number  of  processors, 
implementing  additional 
improvements  to  model 
efficiency,  and  upgrading  model  physics.  The  CHSSI 
work  will  focus  on  interfacing  STWAVE  with 
circulation  and  sediment  transport  models  to  provide 
a  greater  understanding  of  the  physical  processes 
that  impact  directly  on  operations,  equipment,  and 
training. 


“Tests  earlier  this  year 
demonstrated  that  the 
parallel  version  of 
STWAVE  was  over  90 
percent  efficient  using 
32  processors  on  both 
the  Cray  T3E  and  SGI 
Origin  3000.  This 
translates  into  increasing 
the  model  throughput  by 
a  factor  of  28.” 


Using  STWAVE  to  Model  Nearshore 
Waves 

In  the  past,  nearshore  wave  transformation  modeling 
has  been  run  on  HPC  platforms  to  make  forecasts  to 
support  warfighters.  To  meet  stringent  time 
requirements  for  producing  the  forecasts,  the  model 
was  run  with  very  coarse  spatial  resolution  and  less 
frequency  in  time  than  is  optimal.  Under  the  CHSSI 
High  Fidelity  Simulations  of  Littoral  Environments 
project,  STWAVE  was  parallelized  to  exploit  HPC 
capabilities.  Tests  earlier  this  year  demonstrated  that 
the  parallel  version  of  STWAVE  was  over  90  percent 
efficient  using  32  processors  on  both  the  Cray  T3E 
and  SGI  Origin  3000.  This  translates  into  increasing 
the  model  throughput  by  a  factor  of  28.  The  highly 
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Rapid  Transition  to  Operations:  Numerical  Weather 
Modeling 

William  Burnett 

Naval  Meteorology  and  Oceanography  Command  (NAVMETOCCOM),  Stennis  Space  Center,  MS 


The  Primary  Oceanographic  Prediction 
System  (POPS)  produces  and  provides 
critical,  classified  and  unclassified 
atmospheric  and  oceanographic  guidance  to 
Navy  and  DoD  activities  worldwide  on  a  fixed 
schedule,  24  hours  a  day.  POPS  covers  the  entire 
Fleet  Numerical  Meteorology  and  Oceanography 
Center  (FNMOC)  enterprise  including  the 
supercomputing,  communications  (including  receipt 
of  hundreds  of  thousands  of  observations  and 
transmission  of  model  products),  databases,  data 
assimilation  and  distribution,  and  systems  control/ 
monitoring.  It  is  the  engine  that  operates  all  the 
Navy’s  global/regional/tactical  atmospheric, 
oceanographic,  wave,  ice,  and  tropical  cyclone 
models,  and  DoD’s  only  coupled  air/ocean  model. 
The  POPS  is  the  only  national  system  that  assimilates 
classified  and  unclassified  data,  and  produces  and 
disseminates  classified  and  unclassified  global/ 
regional  atmospheric  guidance. 

Meteorological  Data  Assimilation 

Keeping  operational  numerical  weather  models 
“state-of-the-art”  while  transitioning  to  future,  scalable 
computing  architectures  is  a  difficult  but  necessary 
task.  The  computing  industry  is  evolving  into  a  new 
state  in  which  the  preferred  mode  of  large-scale 
computers  is  a  hybrid  system  composed  of  hundreds 
of  nodes,  each  using  tens  of  processors.  Memory  is 
distributed  among  the  nodes,  meaning  that  explicit 
communications  are  needed  for  synchronization  of 
data  during  computations  across  the  nodes.  The 
industry  standard  for  performing  this  communication 
is  known  as  the  Message  Passing  Interface  (MPI). 
Within  each  node,  the  individual  processors  share 
memory  in  a  manner  similar  to  today’s  multi¬ 
processor,  shared  memory  computers,  such  as  the 
Cray  C90.  Communications  among  the  individual 
processors  is  accomplished  through  the  emerging 
standard  OpenMR  However,  the  shared  memory  in 
the  newer  systems  is  cache-based  with  a  relatively 


small  bandwidth,  and  the  processors  are  scalar, 
rather  than  vector  processors.  These  differences, 
and  the  use  of  distributed  memory  across  the  nodes, 
requires  a  significantly  different  programming  system 
than  what  has  been  used  in  the  past  in  order  to 
realize  the  full  potential  of  this  newer  architecture. 

The  Navy  Operational  Global 
Atmospheric  Prediction 
System  (NOGAPS)  and  the 
Coupled  Ocean  Atmosphere 
Mesoscale  Prediction  System 
(COAMPS)  are  complete 
numerical  weather  prediction 
(NWP)  systems.  Each 
receives  raw  meteorological 
observation  data  from  myriad 
sources  around  the  world, 
and  with  a  variety  of  data 
processing  and 
computational  steps,  produces  numerical,  gridded 
“products”  to  satisfy  DoD  requirements  for 
meteorological  support.  This  process  is  known  as 
meteorological  data  assimilation  and  is  typically 
performed  by  marching  forward  in  6-hour  steps 
known  as  “update  cycles.” 

HPC  Resources  Crucial 

The  Oceanographer  of  the  NAVMETOCCOM  and 
FNMOC  agreed  to  a  very  aggressive  schedule  to 
completely  upgrade  the  Navy’s  and  DoD’s  POPS 
supercomputer  from  vector  to  scalable  architectures 
in  1997.  This  schedule  could  not  have  been 
successfully  accomplished  without  DoD’s  HPC 
resources.  MSRC,  CHSSI,  and  PET  are  invaluable 
assets.  They  ensured  that  the  distributed  memory 
systems  purchased  by  the  Oceanographer  of  the 
Navy  in  1999  were  efficient  and  that  the  modeling 
codes  were  optimized  for  the  new  architecture. 
CHSSI  funds  were  leveraged  with  the 
Oceanographer’s  research  and  development  (R&D) 


"The  POPS  is  the  only 
national  system  that 
assimilates  classified 
and  unclassified  data, 
and  produces  and 
disseminates  classified 
and  unclassified  global/ 
regional  atmospheric 
guidance." 
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Figure  1 .  Navy  Operational  Global  Atmospheric  Prediction  System 
(NOGAPS)  previous  1 2-hr  forecast  of  precipitation  rate[mm/12hr 
filled  areas  in  color],  540  and  564  thickness  lines  [dm  -  blue 
contours],  and  sea  level  pressure  [hPa  -  orange  contours]  valid  14 
May  2002  at  00  Zulu 


effort,  HPC  resources  ensured  that  FNMOC 
continued  to  provide  exceptional  service  to  the  Fleet 
and  DoD  other  customers. 

New  Codes  Work  Heterogeneously 

The  new  NOGAPS  and  COAMPS  codes  work  in  a 
heterogeneous  computing  environment  that  is  a 
significant  departure  from  previous  generations. 
Historically,  these  systems  have  run  in  a 
homogeneous  computing  environment  with  all 
modules  running  on  the  same  platform,  e.g.,  a  Cray 
C90.  This  was  possible  because  supercomputers 
such  as  the  C90  delivered  considerable 
computational  power  for  codes  ranging  from  serial  to 
parallel  vector  applications.  The  new  generation  of 
scalable  architecture  supercomputers,  with 
distributed  memories  and  cache-based  processors, 
are  not  general-purpose  machines  and  require  code 
that  works  in  a  heterogeneous  environment. 


funds  to  rapidly  upgrade  and  rewrite  the  NOGAPS 
and  the  COAMPS  codes.  These  funds  were  crucial  to 
ensuring  that  the  updated  codes  were  ready  for  the 
new  systems.  The  CHSSI  code  conversion  effort 
ensured  that  the  Naval  Meteorology  and 
Oceanography  community  could  provide  higher- 
resolution  and  more  accurate  numerical  weather 
products,  in  a  faster  amount  of  time,  for  less  costs. 
Second,  MSRC  resources  (both  DoD  Challenge 
Projects  and  CHSSI)  were  used  to  test  and 
benchmark  the  codes  on  various  machines  to  help 
determine  which  machines  could  efficiently  and 
effectively  operate  the  code.  This  was  crucial  to 
ensuring  that  the  codes  were  ready  for  rapid 
transitions  to  operations.  Third,  PET  resources  were 
used  to  train  both  the  researchers  and  operators  on 
the  new  types  of  HPC  architectures,  and  help  them 
understand  how  to  develop  and  operate  scalable 
systems. 

Visualization  of  atmospheric  data  (both  global  and 
regional)  is  crucial  to  the  scientist  and  the  warfighter. 
Both  NOGAPS  and  COAMPS  outputs  contain  millions 
of  bits  of  information.  Visualization  techniques  helps 
the  user  understand  the  information  and  the  impact 
of  the  environment  on  exercises/operations. 

The  HPC  resources  allowed  the  operational 
community  to  continue  to  provide  seamless 
operational  weather  products  and  information  to  the 
warfighters  during  a  crucial  time.  The  conversion 
from  the  Cray  C90  supercomputers  to  the  SGI  Origin 
3800s  was  an  enormous  and  highly  successful 
undertaking.  During  this  extraordinarily  complex 


Figure  2.  Coupled  Ocean  Atmosphere  Mesoscale  Prediction  System 
(COAMPS)  previous  1 2-hr  forecast  of  precipitation  rate  [mm/1 2hr 
filled  areas  in  color],  540  and  564  thickness  lines  [  dm  -  blue 
contours],  and  sea  level  pressure  [hPa  -  orange  contours]  valid  14 
May  2002  at  00  Zulu 
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3DVAR  —  AFWA's  New  Forecast  Model  Ties 
Research  to  Operations 

Paige  Hughes 

Air  Force  Weather  Agency  (AFWA),  Offutt  AFB,  NE\ 


he  Air  Force  Weather  Agency  (AFWA) 
implemented  an  advanced  observation 
integration  method  September  26,  2002,  that 
significantly  improves  weather  forecast 
model  accuracy. 

The  three-dimensional  variational  data  assimilation 
(3DVAR)  processes  nearly  four  times  the  amount  of 
weather  observations  than  the  previous  method. 
Additionally,  3DVAR  ingests  a  wider  spectrum  of 
observations,  as  many  as  21  various  data  types. 
Developers  contend  that  all  of  the  forecasts 
produced  using  3DVAR  show  significantly  enhanced 
resolution. 

3DVAR  is  capable  of  running  on  45,  15,  and  5 
kilometer  grids  further  enhancing  the  analysis  on 
initial  conditions.  Real-time  verification  of  the  data 
shows  improved  cloud  and  precipitation  locations  of 
the  initial  forecast  time  of  the  model. 

“It’s  a  more  sophisticated  way  of  determining  the 
initial  conditions,  ultimately  leading  to  a  better 
forecast,”  said  Dr.  Jerry  Wegiel,  chief  of  Fine  Scale 
Models  team  at  AFWA  and  the  principle  investigator 
for  3DVAR.  “This  is  a  significant  milestone  for  the 
operational  and  research  community,”  he  added. 

AFWA  is  the  first  in  the  country  to  use  this  method 
operationally  and  with  the  increased  capability,  AFWA 
is  producing  weather  forecast  products  with  greater 
accuracy  for  the  warfighter.  AFWA  maximizes  the 
nation’s  aerospace  and  ground  combat  effectiveness 
by  providing  accurate,  relevant  and  timely  air  and 
space  weather  information  to  the  DoD,  coalition,  and 
national  users. 

3DVAR  is  one  facet  of  a  larger  initiative  to  replace  the 
current  fine  scale  modeling  process  used 
operationally.  As  part  of  this  initiative,  developers  are 
working  on  the  Weather  Research  and  Forecast 
(WRF)  model  slated  to  replace  the  Mesoscale  Model 
Five. 


WRF  is  the  object  of  major  interest  for  meteorologists 
because  it  provides  a  common  framework  for 
research  and  operational  modeling.  WRF 
incorporates  a  portable  modeling  infrastructure  with 
the  latest  atmospheric  science  allowing  researchers 
to  develop  new  numerical  methods  while 
incorporating  real-time  operations.  The  WRF 
initiative  teams  the  nation’s  top  weather  scientists 
from  AFWA,  the  National  Center  for  Environmental 
Prediction,  National  Center  for  Atmospheric 
Research,  and  various  universities;  in  fact,  more  than 
700  scientists  are  involved  in  this  effort. 

AFWA  has  long  been  the 
DoD’s  center  of  excellence 
for  cloud  analysis  using 
high-resolution  satellite 
data,  and  now,  with  the 
implementation  of  WRF- 
3DVAR,  AFWA  is  being 
touted  as  a  leader  in 
forecast  mode  R&DI. 

Funding  for  the  WRF-3DVAR 
initiative  comes  from  the 
HPCMR  The  HPCMP 
provides  the  supercomputer 
services,  high-speed 
network  communications, 
and  computational  science 
expertise  that  enables  the 
Defense  laboratories  and 
test  centers  to  conduct  a  wide  range  of  focused 
research,  development,  and  test  activities.  This 
partnership  puts  advanced  technology  in  the  hands 
of  US  forces  more  quickly,  less  expensively,  and  with 
greater  certainty  of  success. 

AFWA’s  fine  scale  models  team  is  looking  farther 
down  the  research  road  and  has  unveiled  a  concept 


“AFWA  is  the  first  in  the 
country  to  use  this  method 
operationally  and  with  the 
increased  capability,  AFWA 
is  producing  weather 
forecast  products  with 
greater  accuracy  for  the 
warfighter.  AFWA 
maximizes  the  nation’s 
aerospace  and  ground 
combat  effectiveness  by 
providing  accurate, 
relevant  and  timely  air  and 
space  weather  information 
to  the  Department  of 
Defense,  coalition,  and 
national  users.” 
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called  four-dimensional  variational  data  assimilation 
(4DVAR).  With  funding,  4DVAR,  the  program  could 
be  implemented  operationally  as  early  as  2006. 
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CTA:  Climate/Weather/Ocean  Modeling  and  Simulation 
Computer  Resources:  IBM  SP3  [ASC,NCAR,  AWFA], 
SGI  Origin  3000  [ERDC],  and  IBM  SP  PWR  PC  [AWFA] 
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Figure  1 .  The  first  panel  shows  cloud  top  heights  from  an  operational 
MM5  run  using  WRF-3DVAR  as  the  data  assimilation  scheme.  The 
bottom  panel  is  verifying  MB  enhanced  IR  imagery.  (JAAWIN) 
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Forecasting  the  Weather  Under  the  Sea:  NAVO 
MSRC  Scientific  Visualization  Center  Allows  Navy 
to  See  the  Ocean  Environment  for  Undersea 
Recovery  Operations 

Pete  Gruzinskas 

Naval  Oceanographic  Office  (NAVO),  Scientific  Visualization  Center,  Stennis  Space  Center,  MS 


The  ability  to  accurately  predict  environmental 
conditions  is  paramount  to  operational 
consideration  within  the  Navy  and  the 
DoD.  The  recent  Navy  salvage  operation 
(SALVOPS)  of  the  Japanese  fishing  trawler  Ehime 
Maru  provides  an  excellent  example  of  the  need  for 
these  accurate  predictions.  Through  the  application 
of  state  of  the  art  software  analysis  tools  designed  for 
the  analysis  of  R&D  ocean  circulation  models  to  the 
SALVOPS,  the  Navy  was  able  to  successfully 
complete  its  mission. 

On  February  9,  2002,  a  collision  between  a  US  Navy 
submarine  and  the  Ehime  Maru  sent  the  fishing 
vessel  to  the  bottom  of  the  Pacific  Ocean  about  10 
miles  south  of  Diamond  Head,  Oahu,  Hawaii.  The 
decision  was  made  to  retrieve  the  vessel,  which  lay  in 
approximately  1800  feet  of 
water.  A  Crisis  Action  Team 
was  formed,  and  the  NAVO 
MSRC  Visualization  Center 
was  asked  to  provide 
visualization  support  for  the 
Ehime  Maru  retrieval 
operation.  The  first  challenge 
was  to  visualize  the  high- 
resolution  model  output 
generated  by  the  Shallow- 
Water  Analysis  and  Forecast 
System  (SWAFS),  running  on 
the  MSRC’s  IBM-SP.  SWAFS, 
as  shown  in  Figure  1 ,  is  a  3-D 
ocean  circulation  model  that 
produces  time-series  data  containing  ocean  current 
speed  and  direction,  temperature,  and  salinity  fields. 
Initially,  a  2-kilometer  grid  that  covered  the  entire 
Hawaiian  Islands  was  generated  and  used  to  provide 


input  to  a  higher  resolution  (500-meter)  nest,  which 
bounded  the  operation  area. 

Development  of  Analytical  Software 
Environments 

Part  of  the  NAVO  MSRC  Visualization  Center’s 
mission  is  the  development  of  analytical  software 
environments  for  the  DoD  research  community. 

Some  of  these  tools,  designed  to  analyze  ocean 
model  output,  were  modified  to  analyze  the  Navy’s 
operational  model  output  for  the  Ehime  Maru 
SALVOPS.  These  software  tools  provided  significant 
diagnostic  capability,  which  assisted  in  the  validation 
of  the  model  output  in  a  very  complex  environment. 

While  virtual  environments  may  not  provide  all  of  the 
answers  to  data  analysts,  the  fact  of  the  matter  is  the 
real  world  environment  is  3-D.  Features  within  the 
environment  have  a  3-D  structure  that  can  be  difficult, 
if  not  impossible,  to  realize  in  two-dimensional  (2-D) 


Figure  1 .  SWAFS  surface  currents  in  the  operational  area 


“With  the  Navy’s  ever 
evolving  3-D  ocean 
modeling  capabilities 
and  the  matching 
FIPCMP  hardware 
advances,  it  is  now 
possible  to  place  model 
results  on  the  desktop 
of  the  DoD  user  and,  in 
the  case  of  the  Ehime 
Maru  SALVOPS,  on  the 
laptop  of  analysts  in  the 
field." 
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space.  The  same  technology  that  was  built  to  help 
research  and  development  modelers  scrutinize  their 
model  output  was  applied  to  this  operational 
scenario  where  the  speed  and  direction  of  the  ocean 
currents  were  critical  factors. 

An  additional  feature  added  to  the  application, 
unique  to  the  Ehime  Maru  retrieval  operation,  was  the 
display  of  an  Acoustic  Doppler  Current  Profiler 
(ADCP)  buoy.  The  display  of  the  ADCP  data 
provided  analysts  a  direct  comparison  between 
model  output  and  in-situ  current  measurements  in 
near  real-time.  Another  significant  feature  of  the 
virtual  environments  built  for  ocean  model  analysis  is 
portability  —  the  ability  to  run  on  a  variety  of 
hardware  architectures,  including  laptop  computers 
—  which  proved  to  be  critical  in  this  operation. 

Portability  is  accomplished  by  savvy  application  of 
graphics  techniques,  as  well  as  smart  Input/Output 
and  memory  management.  This  ability  was  critical  to 
the  provision  of  an  analysis  environment  and  data  to 
forward-deployed  personnel  on-scene  at  the  Ehime 
Maru  recovery  site.  Other  support  products  were 
developed  to  render  the  recovery  area  in  3-D  from 
high-resolution  bathymetry  (water  depth)  data  to 
delineate  critical  areas  (Figure  2).  Conceptual 
animations  were  built  to  demonstrate  the  mechanics 
of  an  extremely  difficult  recovery  operation. 


Figure  2.  Still  image  showing  Ehime  Maru  retrieval  over  real 
bathymetry  data.  The  animation  was  designed  to  conceptualize/ 
visualize  what  the  recovery  would  look  like  in  the  context  of  the  real 
data  collected  by  the  USNS  SUMNER. 


reconstructed  into  a  virtual  environment  where  a 
variety  of  analytical  techniques  can  be  applied  and 
subsequently  visualized. 

The  NAVO  MSRC  Visualization  staff  was  approached 
before  this  recovery  operation  to  assist  in  the  analysis 
of  the  current  fields  in  the  vicinity  of  the  Ehime  Maru, 
which  lay  at  a  depth  of  approximately  1800  feet.  The 
work  accomplished  in  the  R&D  analysis  arena  was 
directly  applied  to  this  naval  operation.  Techniques, 
such  as  colorized  vectors  and  advected  particles, 
were  employed  to  ensure  that  critical  thresholds  of 
current  speed  were  not  exceeded  in  and  around  the 
salvage  area.  On  scene  (in  situ)  sensors  provided 
near  real-time  measurements,  which  were  displayed 
along  with  the  model  output.  This  was  all  packaged 
and  automated,  to  provide  daily  results  to  analysts’ 
on-scene  in  Oahu. 

A  level  of  confidence  was  established,  based  on 
model  output  and  in  situ  measurements,  that  the 
ocean  currents  in  and  around  the  salvage  area  would 
not  represent  a  threat  to  the  salvage  operation.  The 
virtual  environments  created  by  the  staff  at  the  NAVO 
MSRC  Visualization  Center  provided  a  critical  look 
into  the  environment  around  this  operation,  with 
regard  to  the  undersea  terrain  (bathymetry),  ocean 
current  speed,  direction,  and  temperature.  This  was 
a  successful  SALVOPS  and  the  tools  brought  to  bear 
before  and  during  this  operation  will  serve  both  the 
R&D  and  operational  ocean  modeling  communities. 

Future  3-D  Ocean  Models 

Advances  in  hardware  and  software  technology  have 
helped  close  the  gap  between  computational 
technology  and  the  technology  required  to 
manipulate  and  display  the  results  of  the  3-D  ocean 
model  output.  With  the  Navy’s  ever  evolving  3-D 
ocean  modeling  capabilities  and  the  matching 
HPCMP  hardware  advances,  it  is  now  possible  to 
place  model  results  on  the  desktop  of  the  DoD  user 
and,  in  the  case  of  the  Ehime  Maru  SALVOPS,  on  the 
laptop  of  analysts  in  the  field. 

The  operation  was  successful,  and  all  mission 
objectives  were  accomplished.  This  recovery 
operation  demonstrates  the  excellent  synergy  that 
has  developed  between  the  operational  Navy  and  the 
DoD  research  infrastructure  built  by  the  HPCMP. 


Using  Virtual  Environments  to 
Analyze  Results 

3-D  ocean  circulation  models,  like  those  supported 
by  the  HPCMP,  represent  3-D  grids  with  output  that 
varies  over  time  (four-dimensional).  To  analyze  the 
output  of  these  models  the  computational  space  is 
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Deny  Enemies  Sanctuary 


“Deny  enemies  sanctuary  —  so  they  know  no  corner  of  the  world  is  remote  enough,  no  mountain  high 
enough,  no  cave  or  bunker  deep  enough,  no  SUV  fast  enough,  to  protect  them  from  our  reach. . .  To 
achieve  this  objective,  we  must  have  the  capability  to  locate,  track  and  attack  both  mobile  and  fixed 
targets — any  where,  any  time,  and  under  all  weather  conditions...  To  achieve  this,  we  must  develop 
new  data  links  for  connecting  ground  forces  with  air  support;  new  long-range  precision  strike 
capabilities;  new,  long-range,  deep  penetrating  weapons  that  can  reach  our  adversaries  in  the  caves 
and  hardened  bunkers  where  they  hide...”  —  from  testimony  of  US  Secretary  of  Defense,  Donald 
Rumsfeld,  prepared  for  the  House  Armed  Services  Committee  2003  Defense  Budget  Request, 

February  6,  2002. 

These  words  ring  true.  Recent  events  in  our  nation  only  underscore  their  importance.  As  scientists  and 
engineers  the  message  is  clear.  All  our  efforts  must  be  directed  towards  satisfying  the  Secretary’s 
transformational  goal.  The  following  success  stories  highlight  the  outstanding  accomplishments  of 
DoD  scientists  and  engineers  in  striving  towards  this  goal.  The  transformational  goal  is  actually  a 
continuously  evolving  process  that  requires  constant  innovation  by  the  scientific  community.  The 
ability  to  solve  very  complex  problems  from  quantum  physics  to  high-speed  fluid  flow  is  a  hallmark  of 
these  success  stories.  By  applying  state-of-the-art  algorithms,  coupled  with  high  performance 
computing  resources,  problems  that  could  not  be  solved  as  little  as  5  years  ago  can  now  be  solved 
routinely.  A  selection  of  these  stories  illustrate  the  practical  application  of  modeling  and  simulation  in 
the  design  process  —  with  the  goal  of  producing  better  and  cheaper  weapon  systems  for  the 
warfighter.  Other  stories  reflect  basic  research  efforts  in  materials  modeling  and  device  development. 
Overall,  the  stories  span  many  of  the  HPCMP  computational  technology  areas  including  computational 
fluid  dynamics  (CFD),  computational  structural  mechanics  (CSM),  computational  chemistry  and 
material  science  (CSM),  and  computational  electromagnetics  and  acoustics  (CEA). 

The  first  group  of  stories  in  this  section  highlights  the  use  of  high  performance  computing  in  the  design 
process  for  air,  sea,  and  ground  weapon  systems  applications.  These  stories  include  using  high- 
fidelity  simulations  for  the  design  of  an  advanced  submarine  sail,  end-to-end  simulation  capabilities  for 
projectile-target  interactions,  and  advanced  modeling  and  simulation  of  revolutionary  unmanned  air 
vehicles  (UAV).  These  stories  are  examples  of  the  paradigm  shift  taking  place  within  the  DoD  as  part 
of  the  transformation.  That  is,  developing  cost  effective  weapons  in  a  “virtual”  design  environment. 

The  next  group  of  stories  discuss  materials  and  chemistry  modeling.  The  first  in  this  set  is  a  story  on 
atomistic  modeling  techniques  for  the  development  of  new  energetic  materials.  This  is  followed  by  a 
story  on  materials  modeling  for  device  development.  Next  is  a  story  on  quantum  mechanical 
molecular  modeling  for  the  design  of  advanced  resins  used  in  gun  testing.  Finally  there  is  a  related 
story  on  nanoparticles  used  in  the  design  of  high  energy  density  materials.  In  the  next  group,  there  are 
two  stories  on  modeling  and  simulation  of  weapons  of  mass  destruction.  In  both  stories,  the  authors 
study  the  effects  of  the  non-ideal  airblast  on  the  environment.  The  next  group  of  stories  illustrates  the 
application  of  modeling  and  simulation  in  advanced  sensor  technology  and  advanced  radio  frequency 
(RF)  weapon  technology.  The  first  in  this  group  discusses  modeling  the  performance  of  unattended 
ground  sensors  in  the  modern  complex  battlefield  environment.  It  is  shown  that  high-fidelity 
simulations  are  essential  to  assess  the  performance  limitations  of  next-generation  sensor  technology. 
The  next  story  is  on  modeling  and  simulation  of  foliage  penetrating  radar  and  how  electromagnetic 
models  can  be  used  in  the  development  of  advanced  detection  algorithms.  The  last  story  in  this  group 
discusses  the  use  of  electromagnetic  particle-in-cell  methods  to  design  high  power  microwave 
devices.  These  devices  are  designed  to  defeat  enemy  battlefield  electronics  using  non-lethal  means. 
This  capability  will  be  very  important  in  future  conflicts  where  over-matched  enemy  forces  may  seclude 
themselves  into  the  civilian  population.  The  last  group  in  this  section  shows  examples  of  the 
outstanding  progress  that  has  been  made  in  reactive  flow  modeling.  The  first  in  this  set  discusses 
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multi-phase  CFD  simulations  used  for  next  generation  gun  propulsion  systems.  The  next  is  a  related 
story  on  the  interaction  between  high  temperature  gas  and  solid  propellant  in  tank  and  artillery  gun 
rounds.  The  final  story  in  this  group,  and  in  this  section,  discusses  the  production  of  nanoenergetic 
particles  in  turbulent  reacting  flows,  which  are  expected  to  be  employed  in  next-generation  Army  gun 
systems.  In  summary,  each  of  these  stories  can  relate  to  the  Secretary’s  transformational  goal  on 
denying  enemies  sanctuary  —  whether  it  is  in  unmanned  surveillance  and  tracking  technology 
(sensors,  UAV)  or  improved  lethality  technology  (weapon  platforms,  materials). 

As  the  DoD  leadership  continues  to  transform  military  training  and  doctrine,  it  is  incumbent  upon  the 
scientific  community  to  continue  to  develop  and  deliver  the  best  technologies  to  the  warfighter.  In  the 
end,  these  stories  are  just  a  small  sampling  of  the  outstanding  contributions  made  by  scientists  and 
engineers  across  the  country  in  government  service,  academia,  and  industry.  The  common  thread 
running  throughout  this  national  fabric  is  high  performance  computing.  In  large  measure,  the  work 
represented  by  these  stories  could  not  have  been  accomplished  without  the  significant  investments 
made  in  high  performance  computing  hardware,  software,  and  networking.  These  stories  are  a  tribute, 
and  in  some  sense  a  validation,  of  this  investment. 


Dr.  Anders  Sullivan 

Army  Research  Laboratory,  Sensors  and  Electron  Devices  Directorate,  Adelphi,  MD 
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Design  of  an  Advanced  Submarine  Sail 

Joseph  Gorski 

Naval  Surface  Warfare  Center,  Carderock  Division  (NSWCCD),  West  Bethesda,  MD 


With  submarine  operations  shifting  to 
littoral  missions,  there  is  an  increased 
desire  for  new  systems  that  improve 
littoral  warfare  effectiveness,  and 
designs  that  will  allow  flexibility  for  mission-specific 
reconfiguration  and  rapid  insertion  of  new 
technologies.  Candidate  systems  under 
consideration  include  countermeasure 
enhancements,  high  data  rate  antennas,  unmanned 
underwater  vehicles,  littoral  warfare  weapons, 
advanced  buoyant  cable  antennas,  and  Special 
Operating  Forces  stowage,  among  others. 
Conventional  sails  with  their  low  drag  and  low 
volumes  cannot  accommodate  these  systems.  A 
new  sail  shape,  with  up  to  four  times  the  volume,  is 
necessary  to  enclose  such  new  systems.  This  new 
sail,  which  has  a  dual  curved  canopy-like  shape,  has 
already  been  tested  on  the  US  Navy’s  Large  Scale 
Vehicle  (Figure  1)  with  plans  for  eventual  insertion  on 
the  Virginia  Class  submarine. 

Limited  Experimental  Data 

This  sail  shape  is  a  radical  new  design  for  the  US 
Navy.  There  is  no  historical  database  to  use  as  a 
reference  for  its  design.  Traditionally,  designing  such 
a  new  shape  requires  a  whole  series  of  tests.  Large 
numbers  of  variants  are  built,  tested,  and  evaluated 
encompassing  both  large  and  small  changes.  A 
variety  of  concepts  are  also  explored.  Additionally, 
design  decisions  are  frequently  based  on  limited 
amounts  of  experimental  data  of  a  large  number  of 
geometries. 

Impacting  Design  Decisions 

HPC  environments  bring  a  computational-based 
approach  to  addressing  such  designs.  Many  shapes 
can  be  evaluated  computationally  so  that  only  a  few 
candidate  shapes  are  actually  built  and  tested, 
saving  money.  Computational  studies  can  be 
conducted  in  a  timely  manner,  saving  time  in  the 
design  cycle  over  the  traditional  build  and  test 
procedure.  It  is  only  with  the  large,  parallel,  HPC 


resources  available  in  the 
last  several  years  that  such 
computations  can  be  done 
to  sufficient  accuracy  and 
speed  to  impact  design 
decisions  in  both  the 
preliminary  and  final  stages. 

Unlike  traditional 
experimental  data,  which 
supplies  limited  amounts  of 
data  and  information,  the 
whole  flow  field  can  be 
studied  using  HPC 
resources  and  a 
computational  based 
approach.  This  analysis 

showed  that  large  vortical  flow  structures  were  being 
developed  from  the  early  sail  designs,  causing  a 
large  impact  on  the  propulsor  inflow.  It  was  possible 
to  modify  the  sail  shapes  to  reduce  this  impact  on  the 
propulsor  inflow  once  the  vortical  structures 
phenomena  was  better  understood. 


“High  performance 
computing  allows  many 
shapes  to  be  evaluated 
computationally  so  that 
only  a  few  candidate 
shapes  are  actually  built 
and  tested,  saving  money. 
Another  aspect  of  this  is 
the  timely  manner  in 
which  it  is  now  possible  to 
do  computational  studies 
saving  time  in  the  design 
cycle  over  the  traditional 
build  and  test  procedure." 


Figure  1 .  Advanced  sail  installed  on  the  US  Navy's  large  scale  vehicle 
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Scientific  visualization  has  been  used  to  help 
understand  the  vortical  flow  created  by  the  large 
sails.  The  visualization  was  used  to  help  identify  how 
the  vortices  are  formed  around  the  sail  and  then  flow 
downstream  into  the  propulsor.  A  better 
understanding  of  how  these  vortices  are  formed  has 
led  to  better  ideas  of  how  to  minimize  them  in  the 
redesign  effort. 

Better  Designs  in  Shorter  Time 

Over  the  last  decade,  there  has  been  an  increased 
emphasis  on  the  use  of  computational  tools  to 
evaluate  submarine  flows  and  guide  their  design. 
With  the  advent  of  parallel  computational  capabilities, 
viscous  simulations  have  seen  a  larger  role  in 
predicting  these  flow  fields.  The  shift  to  a  more 
computational  based  design  and  analysis  approach 
leads  to  the  possibility  of  better  designs  in  a  shorter 
amount  of  time.  Such  computations  allow  for  the 
rank  ordering  of  designs  as  well  as  providing  the 
entire  flow  field,  which  can  lead  to  better 
understanding  of  the  flow  physics.  In  the  current 
effort,  modifications  to  candidate  sail  shapes  were 
evaluated  computationally  providing  drag  differences 
as  well  as  propulsor  inflow.  This  allowed  for  the 
development  of  a  new  sail  shape,  which  was  built 
and  tested  on  the  US  Navy’s  Large  Scale  Vehicle — 
with  plans  for  future  implementation  on  the  Virginia 
class  of  submarines.  Such  computations  provide  a 
cost  effective  way  to  evaluate  various  shapes  that 


may  improve  a  design.  In  this  way,  computations  are 
reducing  many  design-oriented  tests.  Future  efforts 
need  to  include  full  propulsor  modeling  as  part  of  the 
calculations.  As  requirements  become  stricter,  future 
design  efforts  will  be  more  sophisticated  involving 
maneuvering  as  well  as  more  complicated  operating 
scenarios.  This  will  require  large  computer 
resources,  as  it  will  be  extremely  difficult  if  not 
impossible  to  perform  such  evaluations 
experimentally. 

References 

Stout,  M.C.  and  Dozier,  D.F.,  “Advanced  Submarine 
Sail,”  Carderock  Division,  NSWC  -  Technical  Digest, 
pp.  58-62,  December  2001 . 

Gorski,  J.J.  and  Coleman,  R.M.,  “Use  of  RANS 
Calculations  in  the  Design  of  a  Submarine  Sail,” 
Proceedings  of  the  NATO  RTO  Applied  Vehicle 
Technology  Panel  Spring  2002  Meeting,  Paris, 

France,  April  2002. 

CTA:  Computational  Fluid  Dynamics 
Computer  Resources:  Cray  T3E  [ARSC] 


-96- 


2002  Success  Stories 


Penetration  into  Deeply  Buried  Structures:  A 
Coupled  Computational  Methodology 

Raju  Namburu  and  Jerry  Clarke 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 


The  ARL  scientists,  working  through  the 
CHSSI,  are  developing  an  end-to-end 
simulation  capability  to  simulate  complex 
projectile-target  interaction  applications. 
This  work  will  accelerate  the  fielding  of  new  weapon 
systems,  improve  armor  technologies,  and  enhance 
protective  structures  technology.  Simulation  of 
penetration  into  deeply  buried  structures  is  the 
primary  targeted  application  area. 


Solving  Individual 
Discipline  Areas 

Typically,  penetration  into 
deeply  buried  structures 
involves  solving  two 
coupled  physics 
applications:  one 
application  is  associated 
with  addressing  the 
modeling  and  simulation  of 
the  penetrator;  the  other 
application  is  associated 
with  the  modeling  and  simulation  of  deeply  buried 
structures.  Most  of  the  earlier  efforts  were  confined 
to  solving  the  individual  discipline  areas  because  of 
the  application’s  complexity  and  computer  resource 
limitations.  Simulating  the  complex  projectile-target 
interaction  in  a  seamless,  coupled,  or 
interdisciplinary  environment  requires  increased  use 
of  HPC  resources. 


"These  preliminary  results 
and  coupled  computational 
environment  show  promise 
for  applicability  to  a  wide 
class  of  penetration-target 
applications  and  practical 
applications  for  fielding 
new  weapon  systems, 
improving  armor 
technologies,  and 
enhancing  protective 
structures  technologies.” 


Fully  Coupling  the  Discipline  Areas 

Penetration  into  deeply  buried  structures  addresses 
development  of  a  scalable-coupled  Eulerian- 
Lagrangian  approach  for  a  wide  class  of  penetration- 
target  applications  in  the  areas  of  weapon 
performance,  weapons  effects,  and  performance  and 
defeat  of  targets.  This  class  is  represented  by 
problems  involving  high-pressure,  high-strain,  and/or 


high-strain-rate  material  interactions,  where  one  or 
more  of  the  material  regions  undergoes  relatively 
small  deformations,  while  the  other  materials  in  the 
problem  undergo  arbitrarily  large  deformations.  The 
penetration  into  earth  problem  is  a  good  example  of 
this,  where  the  penetrator  remains  essentially  elastic, 
while  the  target  can  undergo  very  large  strains  and 
deformations  in  the  region  of  interaction  with  the 
penetrator  (Figure  1).  Nevertheless,  deformation  of 
the  penetrator  can  often  have  a  strong  influence  on 
the  loading  exerted  on  it  by  the  target.  Thus,  a  fully 
coupled  treatment  of  the  penetrator/target 
interactions  and  deformations  is  required  for  these 
applications. 


Figure  1 .  Penetrator  traveling  through  two  different  layers  of  soil 
material  towards  a  deeply  buried  structure 


Simulations  were  performed  of  the  large  deformation 
response  of  a  complex  buried  structure  to  the 
penetrator.  A  buried  structure  using  two  different 
layers  of  clay  and  concrete  was  modeled  using 
shock  physics  software.  The  penetrator  was 
modeled  with  all  the  details  using  large  deformation 
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finite  element  structural  analysis  software.  The 
coupled  simulations  showed  good  scalability  and  the 
results  showed  good  qualitative  agreement.  These 
preliminary  results  and  coupled  computational 
environment  show  promise  for  applicability  to  a  wide 
class  of  penetration-target  applications  and  practical 
applications  for  fielding  new  weapon  systems, 
improving  armor  technologies,  and  enhancing 
protective  structures  technologies. 
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Development  of  a  High-Fidelity  Nonlinear 
Aeroelastic  Solver 

Raymond  Gordnier 

Air  Force  Research  Laboratory  (AFRL),  Wright-Patterson  AFB,  OH 


Correct  prediction  of  the  aeroelastic 
behavior  of  current  and  future  air 
vehicles  remains  a  challenge  for  the  Air 
Force.  The  revolutionary  concepts  being 
proposed  for  a  new  UAV  that  will  be  required  to 
operate  under  extreme  flight  conditions  magnify  the 
problem.  These  new  systems  may  also  incorporate 
new  lightweight  composites  or  have  wing  designs 
that  are  routinely  subjected  to  large  deflections 
during  their  operation.  A  high-fidelity  simulation 
capability  is  required  to  accurately  model  the 
complex,  nonlinear  physical  phenomena  involved  in 
these  types  of  aircraft.  New  generations  of 
multidisciplinary  computational  tools  that  couple 
highly  accurate  CFD  solvers  with  computational 
structural  dynamics  modules  are  required.  The 
incorporation  of  these  types  of  computational 
techniques  into  DoD  weapons  programs  promises 
cost  and  development  time  savings  as  well  as 
improved  operational  performance. 

Linear  Aerodynamic  Techniques 

Traditional  aeroelastic  analysis  has  coupled  linear 
aerodynamic  techniques  such  as  vortex  lattice  and 
panel  methods  with  linear  modal  techniques  for  the 
structural  model.  These  techniques  have  been 
preferred  due  to  their  ease  in  implementation  and 
rapid  solution  times.  While  these  simplified  analysis 
tools  have  been  used  extensively,  they  often  fail  to 
predict  detrimental  aeroelastic  phenomena  since 
they  ignore  important  nonlinear  physical  features  of 
the  problem.  The  simplified  aerodynamic  models 
cannot  address  such  critical  issues  as  transition, 
turbulence,  separation,  buffet,  and  transonic  shocks. 
The  linear  modal  structural  models  are  limited  to 
small  amplitude  responses  and  linear  material 
properties. 


New  Nonlinear  Aeroelastic  Solver 

The  computational  power  available  through  the 
HPCMP  now  allows  higher  fidelity  modeling  to  be 
brought  to  bear  on  the  problem  of  simulating 
aeroelastic  response.  Specifically,  CFD  solvers  may 
replace  linear  aerodynamic  models  for  the  nonlinear 
Euler  and  Navier-Stokes  equations.  Nonlinear 
structural  models  employing  finite  difference  and 
finite  element  techniques  may  be  used  to  represent 
the  structural  response.  Coupling  of  these  enhanced 
techniques  provides  a  powerful  new  nonlinear, 
aeroelastic  simulation  tool  that  can  address 
challenging  fluid-structure  interaction  problems  such 
as  limit-cycle  oscillations,  tail  buffet,  and  transition 
delay  and  drag  reduction  via  compliant  surfaces. 

By  exploiting  the  computer  power  available  through 
the  HPCMP,  the  present  work 
seeks  to  overcome  the 
deficiencies  in  current 
aeroelastic  methods  by 
developing  a  new  fully 
nonlinear  aeroelastic  solver. 

In  this  new  procedure,  a 
higher  order  (2  to  3  times 
more  accurate  than  many 
existing  CFD  codes), 
unsteady,  3-D  Navier-Stokes 
code  is  used  to  simulate  the  aerodynamics  of  the 
problem.  The  structural  dynamics  are  simulated 
using  nonlinear  finite  difference  and  finite  element 
models.  These  independent  solvers  are  coupled  in  a 
synchronized  fashion  to  provide  a  powerful  new 
nonlinear  aeroelastic  tool,  which  has  been  applied  to 
several  problems  that  could  not  previously  be 
simulated  with  classical  linear  techniques. 

One  problem  considered  is  the  direct  numerical 
simulation  of  the  interaction  of  a  transitional  boundary 
layer  flow  with  a  flexible  panel  for  both  subsonic  and 
transonic  flow  speeds  (Figure  1).  At  the  higher  flow 


"The  revolutionary 
concepts  being 
proposed  for  a  new 
UAV  that  will  be 
required  to  operate 
under  extreme  flight 
conditions  magnify  the 
problem.’’ 
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Figure  1 .  3-D  panel  flutter  due  to  interaction  with  a  transitional 
boundary  layer 


speed,  a  traveling  wave  flutter  phenomena  occurs 
that  originates  from  the  coupling  of  the  transitional 
Tollmien-Schlichting  waves  in  the  boundary  layer  with 
higher-mode  flexural  waves  in  the  elastic  panel.  This 
results  in  significant  acoustic  radiation  above  the 
fluttering  panel  (Figure  2).  The  highly  refined  meshes 
(up  to  15  million  grid  points)  and  long  time  histories 
(500,000  time  steps)  required  to  accurately  compute 
this  problem  demand  the  use  of  the  most  powerful 
computing  platforms  available  to  successfully 
complete  this  simulation. 


L.E.  T.E. 


Figure  2. 
Acoustic 
radiation  due 
to  travelling 
wave  flutter  of 
a  flexible 
panel 


This  unique  nonlinear  aeroelastic  tool  has  also 
successfully  captured  the  proper  limit  cycle  behavior 
of  a  delta  wing.  Previous  computations  using  linear 
structural  models  over  predicted  by  four  to  five  times 
the  amplitude  of  the  experimentally  measured  limit 
cycle  response  of  the  delta  wing.  The  inability  of  the 
linear  structural  models  to  simulate  the  actual 
nonlinear  structural  mechanisms  in  the  problem  was 
identified  as  the  cause  of  this  discrepancy.  The 
present  nonlinear  structural  techniques  accurately 
capture  the  correct  amplitude  and  frequency  of  the 
delta  wing  limit-cycle  oscillation  (Figure  3). 


A  novel,  fully  nonlinear,  aeroelastic  solver  has  been 
developed, 
which  exploits 
the  extensive 
computing 
capacity  of  the 
HPCMR  The 
ability  of  this 
new  tool  to 
address  fluid- 
structure 
interaction 
problems  that 
cannot  be 
computed  with 
current  linear  aeroelastic  techniques  has  been 
demonstrated.  This  high-fidelity  simulation  capability 
may  now  be  exploited  to  enhance  the  development 
of  current  and  future  manned  and  unmanned  air 
vehicles. 

Detecting  and  Explaining  Relevant 
Flow  Features 

The  ability  to  effectively  visualize  the  computed 
results  for  the  complex  physical  problems  considered 
in  this  work  is  essential  in  order  to  develop  a 
complete  understanding  of  the  problem  being 
simulated.  The  massive  amounts  of  information 
produced  by  these  high-fidelity  computations  may 
not  be  exploited  without  resorting  extensively  to 
scientific  visualization.  Three-dimensional 
visualizations  of  the  resulting  flowfields  have  been 
used  to  detect  and  explain  relevant  flow  features 
critical  to  the  analysis  of  the  physical  phenomena 
being  investigated.  Furthermore,  the  dynamic  nature 
of  these  problems  could  not  be  discerned  without  the 
production  of  animations  of  the  unsteady  computed 
results.  Without  the  ability  to  produce  these  types  of 
scientific  visualizations,  a  complete  interpretation  of 
these  costly  simulations  would  not  be  possible  and 
much  of  the  computational  effort  would  be  wasted. 

Developing  a  New  Nonlinear 
Aeroelastic  Computational  Capability 

Present  computational  capabilities  employed  to 
analyze  and  predict  the  aeroelastic  behavior  of 
current  and  future  Air  Force  vehicles  rely  almost 
exclusively  on  linear  aerodynamic  and  structural 
models.  These  simplified  models  fail  to  predict  many 
detrimental  aeroelastic  phenomena  since  they  ignore 
important  nonlinear  physical  features  inherent  to  the 
problem.  The  present  research  focuses  on 
developing  a  new,  nonlinear  aeroelastic 
computational  capability  to  overcome  these 


at  the  point  of  maximum  deflection  in  the 
limit  cycle  response 
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deficiencies.  Exploitation  of  this  new  computational 
capability  earlier  in  vehicle  design  and  development 
will  provide  a  means  to  predict  adverse  fluid/structure 
interaction  problems  before  the  flight  test  and 
manufacture  phase  when  redesign  of  the  vehicle  is 
costly.  This  can  provide  significant  cost  and 
development-time  savings  to  an  air  vehicle  program. 
This  type  of  computational  tool  will  also  allow  for  a 
more  rational  design  of  future  UAVs,  which  will  be 
required  to  operate  in  extreme  flight  conditions  where 
nonlinear  aeroelastic  phenomena  will  be  prevalent. 
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Advances  in  defense  weaponry  often 

require  the  development  of  new  energetic 
materials.  These  technological 
advances  often  outpace  the  energetic 
material  development  process,  however,  not  allowing 
for  the  complete  utilization  of  the  new  weapons 
platforms.  It  is  critical  that  the  process  of  developing 
high-performance  energetic  materials  be  optimized 
to  meet  the  needs  of  these  innovative  applications. 
Modeling  and  simulation  must  be  used  in  the 
development  process  where  possible,  eliminating  the 
traditional  costly  and  time-consuming  approaches  of 
formulation  and  experimentation.  With  this  goal  in 
mind,  researchers  at  ARL  have  recently  developed  a 
computational  tool  for  assessing  the  detonation 
performance  of  new  energetic  materials. 

Early  Models  Lacked  Predictive 
Capabilities 

Historically,  theoretical  models  parameterized  to 
experimental  data  were  used  in  simulations  to  aid  in 
analyses  of  experiments.  The  models  were  heavily 
dependent  on  measurements  before  high 
performance  computing.  For  such  models, 
predictive  capabilities  were  extremely  limited.  They 
lacked  the  means  of  discerning  whether 
discrepancies  between  the  theoretical  predictions 
and  the  experimental  measurements  were  a  result  of 
an  inadequate  model  or  an  inaccurate  theory. 

The  Role  of  the  New  Computational 
Tool 

The  role  of  this  method  in  the  development  process 
is  two-fold.  First,  the  method  can  be  used  to  provide 
the  fundamental  understanding  of  energetic  materials 
necessary  for  exploitation  in  the  weapons  system. 
Second,  the  method  can  be  used  to  screen 
candidate  notional  materials  immediately, 
substantially  impacting  time  and  financial 


considerations.  This  computational  tool  can  also  be 
used  to  meet  the  demands  of  advancing  weapons 
technology  that  requires  nano-structured  energetic 
materials.  As  defense  weapons  technology  and 
engineering  continues  to  grow  past  the  current 
understanding  of  the  microscopic-level  phenomena, 
the  implementation  of  this  tool  will  continue  to  fill  in 
existing  knowledge  gaps. 

To  date  the  computational  tool,  exercised  using  the 
resources  of  the  ARL  MSRC,  has  allowed  successful 
prediction  of  the  detonation  properties  for  several 
simple  systems.  With  the 
advances  in  modern 
theoretical  chemical 
modeling  methods, 
highly  scalable  software, 
and  parallel  computer 
architectures,  more 
elaborate,  accurate,  and 
predictive  atomistic 
simulations  are  being 
performed  that  provide 
information  about 
energetic  materials  that  is 
not  amenable  to  direct 
experimental 
observation. 

More  Realistic  and  Complex  Systems 
Simulated 

Future  research  objectives  are  to  simulate  more 
realistic  and  complex  systems,  which  can  only  be 
implemented  through  the  use  of  resources  supplied 
through  the  HPCMP.  The  calculation  of  detonation 
properties  requires  a  multitude  of  simulations  at  a 
variety  of  temperatures  and  pressures,  with  each 
simulation  requiring  millions  of  computations  for 
systems  containing  tens  of  thousands  of  atoms.  As  a 
result,  the  calculation  of  the  detonation  properties  of 


“This  computational  tool  can 
also  be  used  to  meet  the 
demands  of  advancing 
weapons  technology  that 
requires  nano-structured 
energetic  materials.  As 
defense  weapons  technology 
and  engineering  continues  to 
grow  past  the  current 
understanding  of  the 
microscopic-level 
phenomena,  the 
implementation  of  this  tool  will 
continue  to  fill  in  existing 
knowledge  gaps." 
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Figure  1 .  Application  of  the  Reactive  Monte  Carlo  (RxMC)  method  to  the  detonation  of  nitric  oxide  (2N0<  =  >N2+02):  (a)  Detonation  properties  of 
NO;  (b)  Molecular  configuration  snapshot  of  reacting  system  at  P=28.47  GPa  (blue:  N2  molecules;  red:  02  molecules;  green:  NO  molecules) 


a  single  energetic  material  might  require  several  CPU 
weeks.  The  resources  available  through  the  HPCMP 
have  enabled  the  evaluation  of  detonation  properties 
using  the  atomistic  modeling  technique. 
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Ferroelectric  (FE)  materials  have  a  wide  range 
of  applications,  from  the  nonvolatile  memory 
modules  in  computers  to  piezoelectric 
transducers.  In  particular,  lead  zirconate 
titanate  (PZT)  exhibits  a  large  piezoelectric  effect, 
making  PZT  the  primary  active  material  in  naval 
Sound  Navigation  and  Ranging  (SONAR)  devices, 
medical  ultrasound  systems,  and  many  other 
important  applications.  PZT  is  piezoelectric  —  this 
means  that  it  converts  tiny  underwater  pressure 
waves  to  electrical  signals,  or  vice  versa.  The  Navy 
needs  to  understand  PZT  so  that  it  can  develop  new 
materials  with  even  better  properties.  The  next- 
generation  piezoelectric  materials  need  to  be  more 
sensitive  to  small  signals,  more  durable,  and  lighter. 


arrangement  makes 
small,  density 
functional  theory  (DFT)- 
accessible  supercells 
only  an  approximate 
model  of  the  real 
material.  Experimental 
methods  only  probe 
the  ensemble  average 
properties  of  the 
material.  All  phases  of 
PZT  have  extremely  similar  experimental  pair 
distribution  functions  (PDFs),  indicating  similar  local 
structure  despite  different  macroscopic  properties. 


Figure  1 .  ABO,  perovskite  structure 


Due  to  its  technological  importance,  PZT  has  been 
extensively  studied  both  theoretically  with  quantum 
mechanical  studies  of  small  supercells  and 
experimentally  with  methods  such  as  neutron 
scattering  and  X-ray  diffraction.  PZT  assumes  the 
AB03  perovskite 
structure,  (Figure  1). 

However,  the  disorder 
in  the  B-cation 


Thus  the  relationship  between 
the  local  structure  of  the 
material  and  its  superior 
properties  is  not  clear. 

Using  Supercell 
Calculations  to  Study 
Chemical  Composition 

The  focus  of  the  work  has  been 
the  use  of  large,  many  atom 
supercell  calculations  to  study 
the  relationships  between 
chemical  composition,  disorder, 
and  piezoelectricity  in  complex 
oxide  Pb(Zr1  xTix)03  (PZT)  solid  solutions  not 
accessed  with  previous  small  quantum  mechanical 
calculations.  Supercells  of  PZT  that  are  four  to  eight 
times  larger  than  previous  calculations  were 
investigated.  The  large  size  and  the  variety  of  the 
supercells  studied  allowed  probing  of  the  effect  of 
various  atomic  arrangements  on  the  structure  of  the 
material,  something  impossible  to  do  with  the 
ordered  cells  used  in  previous  studies. 

Although  there  are  differences  in  the  macroscopic 
phases  and  polarization  of  various  compositions  of 
PZT,  there  are  many  similarities  within  the  local 
structure  of  each  composition.  The  direction  of  lead 
atom  distortions  in  PZT  governs  the  orientation  of  the 
polarization  and  therefore  the  high  piezoelectric 
response,  which  gives  PZT  its  excellent  SONAR 
properties. 

In  the  calculations,  lead  atoms  mostly  make 
tetragonal  (aligned  with  the  x-axis)  and  orthorhombic 
(with  non-zero  components  along  x  and  y,  or  x  and  z 
axis)  displacements.  The  distortions  are  toward 
titanium  (Ti)  neighbors  and  away  from  zirconium  (Zr) 
neighbors,  conforming  to  the  overall  polarization  as 


"The  properties  which 
make  PZT  effective 
also  make  it 
complicated  to  model. 
The  extensive 
resources  of  the 
HPCMP  enable  us  to 
model  large  enough 
regions  of  the  PZT  so 
that  we  can 
understand  how 
disorder  affects  the 
SONAR  capabilities  of 
PZT." 
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Figure  2.  Projection  of  the  4  x  2  x  1  50/50  supercell  DFT  PZT 
structure  on  the  xy  plane.  The  oxygen  octahedra  are  depicted  by 
diamonds,  and  distortions  from  the  ideal  cubic  perovskite  positions 
are  shown  by  arrows.  Black  for  0,  blue  for  Pb,  green  for  Zr  and  red 
for  Ti.  Pb  atoms  are  1/2  unit  cell  above  the  plane  and  apical  0  atoms 
are  left  out. 


much  as  possible.  Both  preferences  are 
understandable.  Zr  is  a  larger  ion  than  Ti,  which 
means  that  the  purely  repulsive  interaction  between 
Pb  and  B  cations  will  be  stronger  for  Pb-Zr  than  for 
Pb-Ti.  The  conformity  to  overall  direction  of 
polarization  is  due  to  simple  electrostatics,  which 
make  dipole  alignment  favorable.  In  the  case  of  Pb 
atoms  1  and  2  in  Figure  2,  both  driving  forces  can  be 
satisfied.  However,  in  the  case  of  Pb  atoms  3  and  4, 
the  local  preference  to  move  toward  Ti  conflicts  with 
the  desire  to  align  with  the  overall  polarization.  This 
competition  results  in  a  compromise,  with  distortions 
predominantly  along  the  x-direction  for  these  Pb 
atoms. 

The  dependence  of  Pb  distortion  direction  on  the 
local  environment  and  overall  polarization  is 
particularly  important,  as  it  is  responsible  for  the 
compositional  transitions  in  PZT.  In  the  Ti-rich  phase, 
due  to  the  abundance  of  Ti  ions,  a  tetragonal 
displacement  satisfies  the  local  repulsion  energy 
preferences  for  the  majority  of  Pb  atoms.  This  can  be 
seen  from  the  angular  distribution  of  the  Pb 
distortions  (Figure  3).  The  Ti-rich  composition  of  PZT 
has  no  Pb  distortions  with  more  than  a  15-degree 
deviation  away  from  the  tetragonal  ferroelectric 
direction.  The  small  scatter  in  the  direction  of  Pb  off 
centering  gives  rise  to  a  fairly  ordered  phase  with 
large  overall  polarization  along  the  x-axis.  On  the 
other  hand,  in  the  Zr-rich  phase,  Ti  ion  scarcity 
means  that  no  single  direction  can  satisfy  the  local 
energy  preferences  of  all  or  even  most  of  the  Pb 
atoms.  Consequently,  the  Pb  displacement  angles 
are  more  broadly  distributed  between  30  and  80 
degrees  away  from  the  x-axis. 

As  a  result  of  the  disorder  in  the  direction  of  Pb 
displacements,  the  magnitude  of  the  overall 
polarization  is  smaller  for  the  rhombohedral  phase 
than  that  of  the  tetragonal  phase.  The  monoclinic 


phase  is  the  bridge  between  the  rhombohedral  and 
the  tetragonal  phases.  While  many  Pb  ions  can 
satisfy  their  local  energy  preferences  by  making  a 
tetragonal  distortion,  other  Pb  ions  cannot.  The 
displacement  of  the  latter  Pb  ions  will  therefore 
contain  significant  y-axis  and  z-axis  components, 
producing  the  overall  polarization  of  the  monoclinic 
phase. 

Development  of  a  Model  That  Will 
Allow  Study  of  Disordered  Systems 

Crystal  chemistry  (where  atoms  are  regarded  as 
possessing  properties  transferable  between  various 
environments)  provides  an  intuitive  understanding  of 
how  the  atomic  composition  affects  the  structure  and 
the  material  behavior,  closing  the  gap  between  the 
small  unit  cells  available  to  first-principles  theorists 
and  real  disordered  materials  with  complex  B-cation 
arrangements.  The  results  of  the  DFT  calculations 
firmly  establish  the  validity  and  the  accuracy  of  the 
crystal  chemistry  approach  by  revealing  the 
distinction  between  intrinsic  and  the  environment- 
dependent  behavioral  motifs  in  the  material  and 
correlating  the  structural  motifs  with  the  local 
environment  for  each  of  the  constituent  atoms. 

The  existence  of  intrinsic  atomic  behaviors  makes  it 
possible  to  extract  a  simple  intuitive  framework, 
which  explains  both  the  DFT  results  and  the 
macroscopic  compositional  phase  transitions.  This 
framework  can  then  be  made  quantitative  by 
encapsulating  the  chemical  interactions  in  a 
phenomenological,  crystal  chemistry  type  model 
using  well-known  formalisms  to  characterize 
electrostatic  and  bonding  interactions.  This  model 
would  contain  only  physically  intuitive  interactions, 
could  be  applied  to  any  perovskite  material,  and 
would  lend  itself  naturally  to  materials  design. 


Figure  3.  Distribution  of  Pb  distortion  angle  away  from  tetragonal 
direction  for  the  rhombohedral,  monoclinic  and  tetragonal  phases  of 
PZT 
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Such  first-principles  based,  crystal-chemical 
modeling  will  enable  the  design  of  materials  with 
desired  macroscopic  properties  by  relating  the 
variations  in  atomic  composition  of  the  material  to  its 
macroscopic  behavior. 

The  properties  which  make  PZT  effective  also  make  it 
complicated  to  model.  PZT  is  Pb(Zr1  xTix)03,  and  the 
disordered  positions  of  the  Ti  and  Zr  atoms  are 
directly  related  to  the  piezoelectricity.  The  extensive 
resources  of  the  HPCMP  enable  researchers  to 
model  large  enough  regions  of  the  PZT  so  that  we 
can  understand  how  disorder  affects  the  SONAR 
capabilities  of  PZT.  The  plan  is  to  extend  this 
development  to  the  design  of  the  new  piezoelectric 
materials  for  future  Naval  SONAR  needs. 
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Polymer  Modeling  for  Smart  Munitions  Electronics 
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Novel  materials  to  aid  and  promote  the 

adhesion  and  survivability  of  aerodynamic 
measurement  apparatus  in  the  extreme 
operating  conditions  of  gun  launched 
munitions  would  be  beneficial  for  improved  munitions 
performance  and  ultimately  improved  weapon 
lethality.  The  goal  of  this  project  was  to  employ 
quantum  mechanical  molecular  modeling  and 
quantitative  structure-property  relationships  methods 
and  to  design  and  guide  the  synthesis  of  novel  epoxy 
resins  with  enhanced  shock  absorption  properties  for 
use  as  adhesives  in  smart  munitions  electronics. 

Predicting  Bulk  Polymer  Properties 

Quantitative  Structure-Property  Relationships 
(QSPRs),  based  on  molecular  properties  calculated 
using  the  AMI  semiempirical  quantum  mechanical 
method,  have  been  developed  to  predict  bulk 
polymer  properties  without  the  need  for  costly  and 
time-consuming  experimentation.  A  QSPR  model 
was  generated  for  complex  cross-linked  copolymers, 
using  the  regression  analysis  program, 
Comprehensive  DEscriptors  for  Structural  and 
Statistical  Analysis  (CODESSA).  By  applying  an  ad 
hoc  treatment  based  on  elementary  probability 
theory  to  the  QSPR  analysis,  the  ARL  scientists 
developed  a  method  for  computing  bulk  polymer 
glass  transition  temperatures  for  stoichiometric  and 
non-stoichiometric  monomeric  formulations.  A 
model  polymer  —  that  represents  potentially  ideal 
properties  that  can  be  achieved  experimentally  — 
was  synthesized  and  found  to  validate  the  specific 
initial  model  predictions  and  in  general  the 
computational  approach. 

Glass  Transition  Temperature  for 
Polymers 

The  model  was  used  to  predict  the  glass  transition 
temperature  for  a  series  of  polymers  composed  of 
the  widely  used  epoxide,  diglycidyl  ether  of 
bisphenol  A,  cured  with  an  aliphatic  or  aromatic 


amine  curing  agent,  all  having  a  stoichiometric 
formulation  of  reactive  functional  groups.  The 
experimental  data  for  the  glass  transition  temperature 
QSPR  analysis  was  obtained  from  published 
literature  sources.  The  experimental  data  used  to 
validate  the  non-stoichiometric  glass  transition 
temperature  predictions  was  obtained  at  ARL.  For  13 
polymers,  a  designer  correlation  equation  having  a 
coefficient  of 
determination  (R2)  of 
0.9977  was  obtained 
using  four  computed 
molecular  properties. 

The  molecular  properties 
describe  the  propensity 
for  intermolecular 
attractions  and  stiffness  of 
backbone  atoms  of  the 
model  polymer  systems. 

Fundamentally,  it  is  known  that  stoichiometric 
formulations  of  amine-cured  epoxy  resins  give 
maximal  glass  transition  temperature  values  and  that 
for  non-stoichiometric  formulations  the  glass 
transition  temperature  decreases  in  a  manner 
dependent  on  the  extent  of  cross-linking  in  the 
polymer,  which  in  turn,  is  dependent  on  the 
stoichiometric  ratio  of  amine  and  epoxide  functional 
groups.  By  applying  a  treatment  based  on 
elementary  probability  theory  to  molecular  models, 
each  representing  a  different  level  of  cross-linking, 
the  trend  in  experimental  glass  transition 
temperatures  for  non-stoichiometric  formulations  was 
accurately  predicted  (Figure  1). 

Benefits  of  the  Quantitative 
Structure-Property  Relationships 

Two  principle  benefits  are  envisioned.  First  is  a 
revolutionary  tool  to  rapidly  design  and  develop 
polymer  materials  using  robust  computational 
methods  to  keep  pace  with  the  timeline  requirements 
dictated  by  the  FCS  and  the  Objective  Force  Warrior 
by  avoiding  the  need  for  extensive  synthesis  and 


"Quantitative  Structure- 
Property  Relationships 
(QSPRs)  have  been 
developed  to  predict  bulk 
polymer  properties  without 
the  need  for  costly  and  time- 
consuming  experimentation." 
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Figure  1 .  a)  Sample  polymer  structure  used  in  prediction  of  glass  transition  temperatures  for  non-stoichiometric  formulations,  b)  Plot  of 
experimental  and  predicted  glass  transition  temperatures  vs.  stoichiometric  ratio  for  non-stoichiometric  polymer  formulations. 


testing.  The  second  benefit  is  significantly  enhanced 
survivability  and  reliability  of  electronic  components 
for  smart  munitions  that  will  be  enabled  by  the  newly 
developed  polymer  packaging  materials. 
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DoD  researchers  are  reducing  the  size  of 
reactant  particles  such  as  aluminum  (Al) 
to  accelerate  the  burn  rates  of  propellant 
and  explosives.  Microscopic 
understanding  of  the  structure  and  mechanical 
properties  of  encapsulated  Al  nanoparticles  will  have 
significant  impacts  on  the  design  of  high  energy 
density  materials. 

In  recent  years,  there  has  been  a  great  deal  of 
interest  in  the  synthesis  of  nanostructured 
composites  consisting  of  metallic  nanoparticles 
coated  with  a  protective  layer.  Such  nanocomposites 
have  mixed  metallic  and  ceramic  properties  such  as 
high  electric  conductivity  and  high-temperature 
stability.  The  objective  of  this  research  is  to 
investigate  the  oxidation,  structure,  thermodynamics, 
and  mechanical  properties  of  these  encapsulated 
nanoparticles  at  high  temperatures  under  high 
pressures. 

Studying  the  Chemical  Bonding 

between  Dissimilar  Materials 

Interfaces  between  soft/ductile  materials  (metals  or 
polymers)  and  hard  surfaces  (metal  oxides  or 
ceramics)  dominate  many  properties  of  composite 
materials  and  coatings.  The  present  variable-charge 
molecular  dynamics  (VCMD)  study  of  the  oxidation  of 
metallic  nanoparticles  paves  a  way  to  atomistic 
investigations  of  how  chemical  bonding  between 
dissimilar  materials  at  the  atomic  level  determines 
macroscopic  properties  such  as  structure,  adhesion, 
friction,  stiffness,  and  fracture  toughness. 

New  Algorithm  Developed  for  VCMD 
Scheme 

To  describe  chemical  reactions  such  as  oxidation,  the 
VCMD  scheme  developed  by  Streitz  and  Mintmire 
was  employed  in  which  atomic  charges  are 
determined  at  every  time  step  to  equalize 
electronegativity.  This  variable  N-charge  problem 


amounts  to  solving  a  dense 
linear  system,  and  its  solution 
typically  requires  0(N3) 
operations. 

To  perform  multimillion-atom 
VCMD  simulations  on  massively 
parallel  computers,  an  O(N) 
algorithm  was  developed  in 
which  the  fast  multipole  method 
by  Greengard  and  Rokhlin  was 
used  to  operate  matrix-vector  multiplications  in  O(N) 
time  and  the  temporal  locality  was  used  by  initializing 
the  iterative  solution  with  the  previous  time  step’s 
charges.  To  further  accelerate  the  convergence,  a 
multilevel  preconditioning  scheme  was  developed  by 
splitting  the  Coulomb  interaction  matrix  into  short- 
and  long-range  components  and  using  the  sparse 
short-range  matrix  as  a  preconditioner.  This 
algorithm  was  incorporated  into  a  suite  of  scalable 
parallel  molecular  dynamics  algorithms. 

VCMD  Provides  Detailed  Picture 

The  VCMD  scheme  was  successfully  used  to  study 
the  oxidation  dynamics  of  an  aluminum  nanoparticle 
(diameter  200  angstrom).  The  VCMD  simulations 
provide  a  detailed  picture  of  the  rapid  evolution  and 
culmination  of  the  surface  oxide  thickness,  local 
stresses,  and  atomic  diffusivities.  The  VCMD 
simulations  reveal  that  an  aluminum  nanoparticle  in 
an  oxygen-rich  environment  goes  through  a  rapid 
three-step  process  culminating  in  a  stable  oxide 
scale:  i)  Dissociation  of  oxygen  molecules  and  the 
diffusion  of  oxygen  atoms  into  octahedral  and 
subsequently  into  tetrahedral  sites  in  the 
nanoparticle;  ii)  radial  diffusion  of  aluminum  and 
oxygen  atoms  and  the  formation  of  isolated  clusters 
of  corner-sharing  and  edge-sharing  OAI4  tetrahedra; 
and  iii)  coalescence  of  OAI4  clusters  to  form  a  neutral, 
percolating  tetrahedral  network  that  impedes  further 
intrusion  of  oxygen  atoms  into  and  of  Al  atoms  out  of 
the  nanoparticle.  Structural  analysis  reveals  a  40 


"Recent  advances  in 
parallel  computing 
have  made  it  possible 
for  scientists  to 
perform  atomistic 
simulations  of 
materials  involving 
billions  of  atoms.” 
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Figure  1 .  Snapshot  of  an  Al 
nanoparticle  after  466ps  of 
simulation  time.  (A  quarter  of 
the  system  is  cut  out  to  show 
the  aluminum/aluminum-oxide 
interface.)  The  larger  spheres 
correspond  to  oxygen  and 
smaller  spheres  to  aluminum; 
color  represents  the  sign  and 
magnitude  of  the  charge  on  an 
atom. 
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nanoparticle  (see  Figure  1). 

Recent  advances  in  parallel  computing  have  made  it 
possible  for  scientists  to  perform  atomistic 
simulations  of  materials  involving  billions  of  atoms. 

However  rendering  such  large  datasets  with  an 
interactive  speed  is  a  major  challenge.  To  solve  this 
problem,  a  visualization  system  incorporating  parallel 
and  distributed  computing  paradigms  has  been 
developed.  This  system  named  Atomsviewer  uses  a 
parallelized,  fast  visibility-culling  algorithm  based  on 
the  octree  data  structure  to  reduce  the  number  of 
atoms  sent  to  the  graphics  pipeline. 
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SHAMRC  Environment  and  Vehicle  Loads 
Calculations  in  Non-Ideal  Flow 
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High  performance  computing  resources 

have  enabled  scientists  and  engineers  to 
perform  high  fidelity  computational  fluid 
dynamics  calculations  in  support  of  the 
DTRA  Non-Ideal  Air  Blast  (NIAB)  program.  The  NIAB 
program  seeks  to  quantify  the  effects  of  NIAB  on 
military  vehicles  over  a  wide  range  of  geographical 
areas  and  scenarios.  Of  particular  interest  is  the 
ability  to  simulate  computationally  the  environment 
and  vehicle  loads  from  the  exit  jet  produced  by  the 
DTRA  Large  Blast  and  Thermal  Simulator  (LB/TS). 

The  goal  of  this  project  is  the  simulation  of 
environment  and  vehicle  loads  caused  by  non-ideal 
airblasts.  The  ERDC  scientific  visualization  team  was 
presented  with  the  challenge  of  providing  a 
visualization  methodology  and  toolset.  The 
visualization  needs  of  this  project  had  outstripped  the 
capabilities  of  the  commercial-off-the-shelf  tools 
originally  used  as  the  size  and  sophistication  of  the 
computational  simulations  grew.  The  Scientific 
Visualization  Center  (SVC)  team  studied  the 
alternatives  in  view  of  the  growing  requirements,  and 
developed  a  new  approach  that  leveraged  the 
research  team’s  existing  investment  in  a  previously 
developed  OpenGL-based  visualization  application. 
After  collaborating  with  the  research  team  on 
enhancements  to  the  visualization  code,  the  SVC 
team  was  able  to  deliver  a  tool  that  enabled  analysis 
of  the  team’s  data  in  terms  of  vehicle  and 
environment  loads  and  the  dust  entrainment  resulting 
from  a  blast. 

The  Code 

Second-order  Hydrodynamic  Automatic  Mesh 
Refinement  Code  (SHAMRC)  —  pronounced 
shamrock  —  is  a  two-  and  three-dimensional,  finite 
difference,  hydrodynamic  computer  code.  SHAMRC 
is  a  descendant  of  SHARC  (Second-order 
Hydrodynamic  Advanced  Research  Code).  It  is  used 
to  solve  a  variety  of  airblast  related  problems  which 


include  high  explosive  (HE)  detonations,  nuclear 
explosive  (NE)  detonations,  structure  loading, 
thermal  effects  on  airblast,  cloud  rise,  conventional 
munitions  blast  and  fragmentation,  shock  tube 
phenomenology,  dust  and  debris  dispersion,  and 
atmospheric  shock 
propagation. 

SHAMRC’s  capabilities 
and  attributes  include 
multiple  geometries,  non- 
responsive  structures, 
non-interactive  and 
interactive  particles, 
several  atmosphere 
models,  multi-materials,  a 
large  material  library,  HE 
detonations,  a  K-e 
turbulence  model,  and 
water  and  dust 
vaporization.  SHAMRC  is 
second-order  accurate  in 
both  space  and  time  and 
is  fully  conservative  of  mass,  momentum,  and 
energy.  It  is  fast  because  it  employs  a  structured 
Eulerian  grid  and  it  is  efficient  due  to  the  use  of  the 
pre-processor  source  library.  SHAMRC  is  a 
production  research  code  that  currently  has  the 
capability  to  run  in  single-grid  mode  or  with  an 
adaptive  mesh  refinement  (AMR),  both  of  which  have 
been  parallelized  to  take  advantage  of  the  rapidly 
developing  hardware  improvements,  and  ensure  the 
viability  of  the  code.  The  AMR  capability  of  SHAMRC 
allows  the  efficient  calculation  of  certain  classes  of 
problems  that  are  otherwise  intractable  using  the 
conventional  single-grid  method. 

SHAMRC  has  been  parallelized  to  run  on  large 
scalable  parallel  platforms.  It  uses  the  Message 
Passing  Interface  library  to  achieve  parallelism.  Initial 
versions  of  the  code  were  parallelized  in  3-D  only  and 
the  Eulerian  grid  was  decomposed  in  only  one 


“SHAMRC  is  used  to  solve  a 
variety  of  airblast  related 
problems  which  include  high 
explosive  (HE)  detonations, 
nuclear  explosive  (NE) 
detonations,  structure 
loading,  thermal  effects  on 
airblast,  cloud  rise, 
conventional  munitions  blast 
and  fragmentation,  shock 
tube  phenomenology,  dust 
and  debris  dispersion,  and 
atmospheric  shock 
propagation.” 


-  Ill  - 


-  Deny  Enemies  Sanctuary 


dimension.  The  calculations  used  this  parallel 
model.  Since  that  time,  the  code  has  been  modified 
to  perform  automatic  domain  decomposition  on  the 
Eulerian  grid  in  up  to  three  dimensions  for  both  the 
2-D  and  3-D  versions  of  the  code.  The  parallel 
version  of  SHAMRC  has  been  tested  on  several 
parallel  platforms 

There  are  several  examples  of  the  effects  of  using 
these  new  codes.  Figure  1  shows  airblast  loads  on 
an  Ml  1 0-A2  self-propelled  howitzer  located  in  the  test 
section  exterior  to  the  LB/TS.  The  surface  is  colored 
by  the  pressure  from  the  blast  wave.  Figure  2  is  a 
close-up  of  the  howitzer.  Figure  3  is  a  wider  view  of 
the  howitzer  on  the  test  platform  at  the  same 
calculational  time.  Both  images  show  the  interaction 
of  the  dust  with  the  vehicle.  The  presence  of  the 
dust  significantly  modifies  the  airblast  environment 
and  the  loads  on  the  vehicle. 

CTA:  Computational  Fluid  Dynamics 
Computer  Resources:  SGI  Origin  3000  [ERDC] 


Figure  1 .  Isosurface  of  dust  density  interaction  with  the  Ml  10-A2 
self-propelled  howitzer 


Figure  2.  Isosurface  of  energy  interacting  with  Ml  10-A2  self-propelled 
howitzer 


Figure  3.  Wide-angle  view  of  isosurface  of  dust  denisty  interacting  with 
M110-A2  self-propelled  howitzer 
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Modeling  Explosions  in  Buried  Chambers  and 
Tunnel  Systems 

John  Bell,  Marc  Day,  Allen  Kuhl,  and  Charles  Rendleman 
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Countering  the  proliferation  of  weapons  of 
mass  destruction  (i.e. ,  chemical,  biological 
or  nuclear  weapons)  is  a  critical  mission 
area  for  DTRA.  Assessment  of  explosion 
effects  related  to  military  scenarios  provides 
important  inputs  to  military  strategy.  For  example,  if 
an  underground  bunker  filled  with  chemical/ 
biological  (C/B)  weapons  is  attacked,  DTRA  must 
predict  the  consequences.  Scientists  must  address 
issues  like  the  evolution  in  space  and  time  of  the 
energy  released  during  the  explosion,  the  risk  that 
the  energy  released  will  destroy  the  structure,  the 
likelihood  of  C/B  weapons  breaching  containment, 
and  the  possibility  of  destroying  the  agent  from  the 
high-temperature  environment  of  the  explosion. 

Other  questions  such  as  how  much  of  the  agent  will 

be  ejected  from  the 
bunker  and  how  the 
agent  is  dispersed  in  the 
atmosphere  (at  what  risk 
to  civilians)  must  also  be 
addressed. 

Scientists  at  DTRA  are 
developing  the 
computational  tools 
required  for  detailed 
assessment  of  such 
explosions.  By  using  the 
advanced  computer 
architectures  available  in 
the  DoD  HPCMP, 
combined  with 
sophisticated  adaptive 
numerical  methodology,  it  is  now  possible  to  perform 
detailed,  end-to-end  numerical  simulations  of  the  3-D 
time-dependent  explosion  field. 


Physical  Modeling  Time  is 
Consuming 

Traditionally,  physical  modeling,  either  in  reduced 
scale  laboratory  settings  or  in  full-scale  field 
experiments,  has  been  used  to  answer  the  questions. 
Such  experiments  are  time  consuming  and 
expensive  compared  to  the  results  that  could  be 
obtained  using  numerical  simulations.  Furthermore, 
experiments  are  difficult  to  instrument  so  high  quality 
data  is  hard  to  obtain. 

Computational  Models  Overcome 
Traditional  Limitations 

HPC  is  an  enabling  technology  for  the  simulations 
that  have  been  undertaken.  Chemically  reacting 
compressible  flows  are  basically  intractable  in  three 
dimensions  without  the  capabilities  provided  by  HPC 
systems.  Typical  models  considered  in  the 
calculations  involved  6  chemical  species  with  10-20 
million  computational  zones. 

HPC  overcomes  the  traditional  limitations  by 
permitting  the  scientist  to  make  predictions  on  the 
fate  of  C/B  agents  using  computational  models  rather 
than  the  more  costly  physical  model  experiments. 

The  use  of  HPC  allows  DTRA  researchers  to  assess 
different  scenarios  based  on  physics-based 
predictive  modeling  rather  than  relying  on  heuristic 
scaling  laws  and  phenomenological  models. 

HPC,  as  applied  with  high-order  numerically  accurate 
AMR  simulations,  has  made  it  possible  to  provide 
predictions  of  the  rate  of  combustion  of  C/B  agents  in 
a  closed  chamber  (Figure  1).  The  results  of  the 
numerical  simulations  have  been  validated  against 
physical  scale  model  experiments.  It  is  possible  for 
the  weapons  effects  specialist  to  evaluate  numerically 
the  influence  of  the  size  of  the  explosive  charge, 
explosive  placement,  and  C/B  agent  configuration. 


“HPC  overcomes  the 
traditional  limitations  by 
permitting  the  scientist  to 
make  predictions  on  the  fate 
of  chemical/biological  agents 
using  computational  models 
rather  than  the  more  costly 
physical  model  experiments. 
The  use  of  HPC  allows  DTRA 
researchers  to  assess 
different  scenarios  based  on 
physics-based  predictive 
modeling  rather  than  relying 
on  heuristic  scaling  laws  and 
phenomenological  models.” 
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Fuel  Consumption  vs.  Charge  Size  and  Location 


O  =  Reflected  shock  interaction 


0.2g,  17cm 


0.2g,  10cm 


0.5g,  10cm 


Reflected  Shock  Interaction 


Figure  1 .  Simulated  fuel  consumption  and 
temperature  profiles  for  a  series  of  small-scale  (1 .75) 
experiments  used  to  model  a  single  bay  of  the 
Dipole  Jewel  Test  Field.  In  the  experiments,  a  small 
explosive  charge  detonated  inside  a  closed 
container  induces  turbulent  mixing  and  shock- 
induced  combustion  of  a  surrogate  for  a  biological 
agent.  The  resulting  combustion  characteristics 
depend  strongly  on  the  size  of  the  explosion  charge, 
and  its  proximity  to  the  combustible  material.  The 
figure  depicts  typical  time  histories  for  three  cases  of 
varying  charge  size  (0.2-0.5g)  and  separation 
distance  (10-1 7mm).  To  the  right,  the  temperature 
field  is  shown  for  the  vertical  midplane  for  the  side  of 
the  vessel  containing  the  combustible;  at  the  instant 
that  reflected  blast  waves  interact  with  the  fuel. 

When  the  detonation  is  far  from  the  initial  fuel,  very 
little  combustion  is  observed,  even  over  long  times. 

If  the  charge  is  closer,  the  incident  blast  wave 
induces  sufficient  turbulent  mixing  of  the  fuel  and  air 
and  the  waves  reflected  from  the  rear  wall  ignite  the 
mixture.  For  stronger  explosions,  the  incident  shock 
initiates  more  vigorous  volumetric  combustion. 


Scientific  visualization  is  an  important  part  of  the 
work.  The  simulations  result  in  large  data  sets 
containing  the  details  of  chemical  and  explosive 
reactions  in  complex,  turbulent  flow-fields.  The  only 
effective  way  to  understand  and  extract  information 
from  this  data  is  to  use  visualization  tools. 

Extending  the  Adaptive  Mesh 
Refinement  Modeling  Software 

The  computations  performed  to  date  provide  a 
compelling  qualitative  picture  to  explain  the 
observations  obtained  during  field  experiments. 
Based  on  the  successes  achieved  with  the  AMR 
modeling  of  confined  explosions  and  the  initial  work 
with  the  simulation  of  thermobaric  explosives,  the 
DTRA  scientists  intend  to  extend  the  capabilities 
supported  by  the  software.  The  extended  version  of 
the  software  will  provide  end-to-end  simulation 
capabilities  for  the  modeling  of  thermobaric  explosive 
devices  in  chambers  and  tunnel  complexes  by 
adding  the  AMR  solid-mechanics  and  low-Mach 
number  reactive  Navier-Stokes  modeling  capability  to 
the  present  reactive  Navier-Stokes  software. 

Future  enhancements  to  the  modeling  package  are 
intended  to  remove  some  of  the  limitations  in  the 
physical  assumptions  made  in  the  simulations,  for 
example,  by  including  the  effects  of  radiation  in 
coupled  equations  of  motion.  Another  improvement 
will  provide  for  venting  of  the  combustion  and 
explosion  products  into  the  background  environment 


in  order  to  track  and  quantify  the  release  of  C/B 
materials.  Finally,  DTRA  scientists  will  improve  the 
simulations  by  examining  other  explosive  agents, 
such  as  the  new  classes  of  thermobaric  explosives. 
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Comprehensive,  reliable,  situational 

information  is  imperative  for  the  success 
of  light-armor,  maneuver  dominated 
Future  Combat  System  operations. 
Tactically  significant  terrain  includes  large-scale 
physiographic  features  such  as  forests,  hills,  passes, 
narrow  valleys,  or  rivers.  These  complex  battlefield 
environments  are  extremely  difficult  sensor  settings. 
Opposing  forces  will  adapt  their  counter  operations 
to  exploit  poor  sensor  coverage  circumstances.  In 
these  complex  settings/contexts,  passive  seismic  and 
acoustic  Unattended  Ground  Sensors  (UGSs)  will 
provide  unique  non-line-of-sight  (NLOS)  and  beyond- 
line-of-sight  (BLOS)  information  under  conditions  that 
are  poorly  covered  by  air  breathing  or  spaced  based 
sensor  platforms.  The  NLOS  attribute  is  a  result  of 
readily  “bending”  signal  wavefronts  as  they 
propagate  through  geologic  and  atmospheric  media. 
Unfortunately,  the  inherent  variability  of  terrain  and 
meteorology  also  leads  to  large  fluctuations  in  signal 
characteristics  and  consequently  severe 
degradations  in  information  accuracy  and  reliability. 
These  environmental  and  terrain  induced  information 
degradations  can  be  mitigated  by  deploying  high 
population  sensor  networks,  by  calibrating  each  UGS 
node  to  its  specific  setting,  and  by  fusing  diverse 
sensor  information  with  optimized  algorithms. 

In  recent  field  trials,  seismic  sensors  have 
demonstrated  their  applicability  to  target  bearing, 
range,  and  classification  problems  at  target  ranges 
well  over  1000  m.  However,  seismic  sensors  have, 
historically,  not  been  heavily  relied  upon  in  practical 
tactical  systems.  This  is  largely  due  to  the  strong 
effects  of  geology  on  the  character  of  seismic  data 
and  the  highly  variable,  unknown  geological 
characteristics  of  each  deployment  setting.  Before 
seismic  sensors  can  reach  their  full  potential  in 


fielded  systems,  methods  must  be  developed  that 
address  the  geologic 
variability  issues.  Geologic 
adaptation  is  the  central 
demonstration  for  this 
project.  The  team  also 
addressed  the  more 
general  problem  of  how 
target-track  data  fusion  for 
a  large  number  (greater 
than  6)  of  spatially 
distributed  UGS  sensors 
can  be  done  and  the 
consequent  improvement 
of  tracking  accuracy.  Field 
demonstrations  with 
comparable  numbers  of 
physical  hardware  are  not 
anticipated  for  several 
more  years  and  as  a 
consequence,  optimized  UGS  data-fusion  methods 
have  not  been  previously  given.  It  is  only  through  the 
use  of  high  fidelity  simulations  that  we  are  able  to 
develop  these  new  adaptation  and  tracking  methods. 

Current  Focus 

The  current  focus  of  the  modeling  effort  includes  (1) 
simulations  of  armored  tracked  vehicles  moving  over 
open  terrain  with  realistic  topography  and  subsurface 
geology,  (2)  the  effects  of  topography  and  geology 
on  the  vehicle  signatures,  (3)  high-fidelity  modeling  of 
tracked  vehicle  dynamics  and  their  track  pad  forces 
so  that  these  forces  can  be  used  as  inputs  to  seismic 
simulations,  and  (4)  accuracy  validation  by 
comparisons  with  field  test  data.  Because  tracked 
vehicles  generate  target-specific  seismic  signatures 
at  ranges  of  roughly  1  km,  models  on  the  scale  of  1 


“This  capability  supports  the 
advanced  sensor  network 
systems  envisioned  for  the 
Objective  Force  by 
demonstrating  the  target 
tracking  and  classification 
capabilities  of  seismic 
sensor  networks  for  non- 
line-of-sight  and  beyond- 
line-of-sight  situational 
awareness  and  by  providing 
developers  of  tactical 
systems  with  a  database  of 
seismic  signatures  for  a 
wide  variety  of  geologic 
conditions.” 
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Figure  1 .  Results  from  a  seismic  simulation  of  a  tank  moving  over  complex  open  terrain 


km2  in  surface  area  were  simulated.  The  resulting 
synthetic  signatures  allow  development  of  sensor 
system  algorithms  that  fully  exploit  seismic  signals. 
Furthermore,  this  capability  supports  the  advanced 
sensor  network  systems  envisioned  for  the  Objective 
Force  by  demonstrating  the  target  tracking  and 
classification  capabilities  of  seismic  sensor  networks 
for  NLOS  and  BLOS  situational  awareness  and  by 
providing  developers  of  tactical  systems  with  a 
database  of  seismic  signatures  for  a  wide  variety  of 
geologic  conditions. 

The  ability  to  simulate  high  fidelity,  time-varying, 
seismic  wavefields  that  include  realistic  signal 
complexity  is  central  to  the  team’s  demonstration  of 
seismic  sensor  geologic  adaptation  and  to  the 
development  of  optimized  track-location  information 


fusion  methods.  It  is  particularly  important  that  the 
signal  complexity  include  the  dynamic  harmonic 
energy  shifts  characteristic  of  moving  targets,  and  the 
effects  of  strong  geologic  contrasts  (large  variations 
in  amplitude,  coherence,  and  wavefront  curvature). 

Geologic  Variations 

Geology  can  be  highly  variable  across  geographic 
regions  and  within  a  specific  location.  These 
variations  have  first  order  effects  on  the  character  of 
seismic  signals  and  must  be  compensated  for  before 
seismic  sensors  can  be  employed  in  battlefield 
systems.  For  basic  seismic  system  operation,  only 
two  geologic  parameters  are  required.  These  include 
the  propagation  speed  of  the  incoming  seismic 
surface  waves,  and  the  rate  of  decay  of  surface 
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waves.  Using  simulated  data  shows  that  these  basic 
properties  are  easily  derived  from  simple  calibration 
methods.  It  further  demonstrates  that  target  tracking 
information  can  be  corrected  allowing  compensation 
for  very  complex  signal  effects  resulting  from  strong 
geologic  contrasts.  This  demonstrates  that  target 
tracks  to  moving  vehicles  are  substantively  improved 
by  application  of  the  correction  functions  and 
geology  parameters  derived  from  the  calibration 
events. 

Figure  1  (a-d)  shows  an  example  resulting  from  a 
simulation  of  a  tank  moving  over  complex  open 
terrain.  Seismic  sources  were  taken  from  a  vehicle- 
dynamics  simulation  of  the  tank.  Figure  1  a  and 
figure  1  b  show  snapshots  of  ground  vibration 
amplitudes  as  the  tank  moves  toward  and  away  from 
a  wide  trench.  Figure  1  c  shows  the  speed  of  the 
tank,  and  figure  Id  shows  a  synthetic  seismic 
signature  spectrogram  of  the  ground  vibration  near 
the  middle  of  the  model  surface.  The  signature 
contains  realistic  features  apparent  in  field 
measurements,  such  as  multiple  harmonics  and 
evolution  of  the  spectral  content  with  vehicle  speed. 
The  series  of  overlaying  black  lines  show  expected 
harmonics  from  tank  wheels  rolling  over  track  blocks. 

Fusing  Network  Nodes 

We  fuse  line-of-bearing  (LOB)  and  range  information 
from  each  node  in  the  network  with  two  methods. 

The  simplest  is  an  outlier  rejection  method  based  on 
the  mean  and  standard  deviation  of  each  individual 
nodes  location  estimate.  The  second  approach  uses 
an  optimum  non-linear,  weighted,  least-squares  error 
minimization  (WLS)  with  weights  determined  from  the 
information  (LOB  and  range)  variance.  Both  these 
approaches  are  shown  to  give  comparable 
performance.  The  WLS  approach  is  expected  to  be 
more  appropriate  with  higher  network  node 
populations  and  when  considering  more  diverse 
sensor  inputs. 


Seismic  UGSs  networks  can  be  adapted  to  specific 
geologic  context  by  using  a  sparse  sequence  of 
calibrated  events.  All  that  is  required  in  the 
suggested  calibration  method  is  a  consistent  source 
excitation  mechanism  and  meter  scale  source 
position  accuracy.  These  simple  criteria  can  be 
easily  met  in  a  wide  variety  of  ways,  including 
monitoring  the  seismic  signals  generated  by  the 
network  deployment  vehicle.  The  geologic 
adaptation  functions,  derived  from  the  calibration 
data,  are  then  applied  to  the  moving  target  LOB  and 
range  estimates  for  each  UGS  node  in  the  network. 
The  results  show  that  the  adapted  and  fused  moving 
vehicle  network  track  results  smoothly  converge  to 
errors  as  small  as  2  m.  The  surprising  correlation 
between  the  location  calibration  errors  and  the 
vehicle  tracking  error  indicate  that  the  network 
performance  might  be  quantifiable  at  the  time  of 
deployment.  This  would  have  broad  practical  utility 
in  designating  target  engagement  points  and 
weapons  systems. 

CTA:  Computational  Electromagnetics  and  Acoustics 
Computer  Resources:  Cray  T3E[AHPCRC,  ERDC] 
and  SGI  Origin  3000  [ERDC] 
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Identification  of  "Hot-Spots”  on  Tactical  Vehicles 
at  UHF  Frequencies 

Anders  Sullivan 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 
Lawrence  Carin 

Department  of  Electrical  and  Computer  Engineering,  Duke  University,  Durham,  NC 


Within  the  DoD,  there  is  considerable 
interest  in  evaluating  the  use  of  low 
frequency,  ultra-wideband  (UWB) 
imaging  radar  to  detect  tactical  vehicles 
concealed  by  foliage  and  targets  buried  in  the 
ground.  At  low  frequencies,  the  tree  canopy  is 
essentially  transparent.  Likewise,  low-frequency 
radar  can  efficiently  propagate  through  the  ground 
for  subsurface  target  detection.  Despite  these 
advantages,  the  development  of  robust  target 
detection  algorithms  has  been  hampered  by  the 
complexity  of  the  scattering  physics  due,  in  part,  to 
the  large  number  of  physical  parameters  involved 
and  the  frequency  diversity  inherent  in  UWB  systems. 
The  objective  of  the  CHSSI  CEA-9  project  is  to 
develop  high  performance  software  for  predicting 
electromagnetic  scattering  from  general  3-D  surface 
and  subsurface  targets.  In  particular,  attention  was 
directed  toward  radar-based,  wide-area  surveillance 
of  buried  mines,  unexploded  ordnance  (UXO), 
bunkers,  tunnels,  and  large  ground  vehicles. 
Validated  electromagnetic  models  will  permit  the 
analysis  of  many  target  scenarios  at  a  fraction  of  the 
cost  of  a  measurements  program. 

Multi-level  Fast  Multipole  Algorithm 
Developed 

The  software  developed  in  this  project  is  based  on 
the  fast  multipole  method  (FMM).  The  FMM 
constitutes  a  recently  developed  extension  to  the 
MoM  technique.  In  the  MoM,  the  unknown 
constituents  in  an  integral  equation  (e.g.,  surface 
currents)  are  represented  in  terms  of  a  set  of  known 
expansion  functions  with  unknown  coefficients.  The 
MoM  results  in  the  matrix  equation  Zi  =  v,  where  Z  is 
an  N  x  N  “impedance”  matrix,  v  is  an  N  x  1  vector 


representing  the  incident  fields,  and  /'  is  an  N  x  1 
vector  representing  the  N  unknown  expansion- 
function  coefficients.  While  the  MoM  is  a  very 
powerful  and  general  technique,  it  has  limitations  that 
constrain  its  use.  In 
particular,  one  must  store 
the  matrix  Z,  and 
therefore,  computer 
memory  limits  the 
number  of  N  unknowns 
that  can  be  considered. 

Consequently,  the  need 
to  store  the  N  x  N  matrix  Z 
translates  to  a  limitation  on  the  electrical  size  of  the 
target  that  can  be  considered  with  a  given  computer 
memory.  To  mitigate  this  problem,  the  ARL  team  has 
developed  a  fully  scalable,  multi-level  fast  multipole 
algorithm  (MLFMA),  which  can  be  applied  to  surface 
(tanks,  trucks,  etc.)  or  subsurface  (mines,  bunkers, 
etc.)  targets.  The  MLFMA  is  a  technique  in  which  the 
expansion  functions  are  grouped  into  M  clusters  and 
the  interactions  are  treated  globally,  cluster  by 
cluster.  The  MLFMA  generally  requires  0(N\ogN)  in 
runtime  and  memory,  while  the  MoM  requires  0(A/2) 
in  runtime  and  memory.  To  model  these  large  targets 
where  the  number  of  unknowns  may  approach  one 
million,  high-performance  computing  resources  are 
required.  The  parallel  MLFMA  software  was 
developed  to  run  on  the  MSRC  SGI  and  IBM  HPC 
platforms. 

Validating  Electromagnetic  Models 

The  ARL  has  been  developing  low  frequency  UWB 
radar  technology  for  detection  of  foliage-concealed 
targets  for  many  years.  Based  on  recent  field  test 
data,  researchers  know  that  the  UWB  radar  has  the 


“Validated  electromagnetic 
models  will  permit  the 
analysis  of  many  target 
scenarios  at  a  fraction  of  the 
cost  of  a  measurements 
program.” 
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potential  to  “see”  foliage-concealed  targets.  The 
question  remains,  however,  on  whether  they  can 
develop  an  understanding  of  the  underlying 
signature  and  whether  it  is  robust  and  reliable 
enough  to  be  militarily  significant.  Rigorous 
electromagnetic  models  such  as  the  MLFMA  can 
help  answer  this  question.  As  an  example,  the  ARL 
team  computed  the  vertical  transmit-vertical  receive 
(VV)  envelope  response  from  a  T-72  for  nose-on 
incidence  (0  =  55°,  <p  =  0°)  using  a  UHF  impulse.  The 
UHF  pulse  spectrum  had  a  bandwidth  of  246^34 
MHz.  This  bandwidth  corresponded  to  a  range 
resolution  of  0.8  m.  As  shown  in  Figure  1 ,  the  time- 
domain  response  was  then  transformed  into  the 
space  domain  and  mapped  onto  the  surface  of  the 
tank.  This  was  done  in  order  to  correlate  the  peak 
returns  in  the  time  waveform  impulse  response  to 
specific  physical  features  on  the  target  —  in  this 
case,  the  front  fender  area,  cupola,  and  rear  gas 
tank.  These  highlighted  (red)  areas  are  the  local 
scattering  centers  (or  “hot  spots”)  on  the  T-72  for  this 
particular  angle  of  incidence.  This  analysis  shows 
that  the  response  from  the  target  is  directly  linked  to 
the  shape  and  orientation  of  the  vehicle.  This  aspect 
of  target  scattering  may  be  useful  for  ATR  algorithm 


development.  In  the  end,  it  may  be  the  key  to 
discriminating  targets  of  interest  such  as  tanks, 
trucks,  missile  launchers,  etc.,  from  the  large  clutter 
sources  (trees). 
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Figure  1 .  Impulse  response  mapped  on  T-72 
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Simulation  and  Redesign  of  the  Compact  A6 
Magnetron 
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Radio  frequency  (RF)  weapons  will  be 
designed  to  work  against  hostile 
battlefield  electronics.  Given  the 
importance  of  electronics  in  the  modern 
battlefield,  the  ability  to  target  these  electronics  will 
give  the  US  a  distinct  advantage  against  technically 
advanced  adversaries.  The  effect  of  RF  radiation  on 
the  electronics  can  vary  from  transient  disruption  to 
destruction  of  hardware,  depending  on  the  power 
and  frequency  of  the  radiation.  Furthermore,  since 
only  electronic  equipment  is  affected,  RF  radiation 
can  be  applied  in  a  non-lethal  manner,  giving  the 
warfighter  a  broader  range  of  options  for  dealing  with 
opponents  of  varying  sophistication  and  strength. 

This  generation  of  electromagnetic  radiation  is  critical 
to  the  DoD’s  advanced  RF  weapon  effort. 

Predicting  High  Power  Microwave 
Performance  Using  ICEPIC 

Improved  Concurrent  Electromagnetic  Particle-in-Cell 
(ICEPIC)  predicts  the  performance  of  several  high 
power  microwave  (HPM)  devices  in  current 
development  for  the  US  Air  Force.  With  an  accurate 
simulation  tool,  devices  such  as  the  relativistic 
klystron  oscillator  (RKO),  the  magnetically  insulated 
line  oscillator  (MILO),  the  relativistic  magnetron,  and 
high  frequency  gyrotrons  can  be  improved.  ICEPIC 
simulates  the  electrodynamics  (Maxwell’s  equations) 
and  charged  particle  dynamics  (relativistic  Lorenz’s 
force  law)  with  second-order  time  and  space 
accuracy  in  HPM  sources.  These  simulations 
accurately  predict  RF-production  and  antenna  gain 
compared  with  experimental  testing.  In  general, 
ICEPIC  development  has  been  driven  by  the  desire  to 
achieve  high  power  in  a  compact  RF  device.  This 
implies  that  the  device  is  operated  at  high  current 
and  high  space  charge.  Consequently,  these  devices 
are  highly  non-linear  and  have  complex  three- 
dimensional  geometries  with  small  features.  ICEPIC 


may  be  used  both  for  fine-tuning  well-understood 
devices  and  for  testing  new  designs. 


Virtual  Prototyping  with  ICEPIC 

Researchers  at  the  AFRL  are  performing  virtual 
prototyping  of  RF  systems,  from  pulse  power  drivers 
to  beam  sources  through  the  antennas,  with  the 
ICEPIC  HPC  software.  They  have  developed  ICEPIC 
over  the  past  several  years  with 
AFSOR  funding.  This  code 
simulates  the  electrodynamics 
(Maxwell’s  equations)  and 
charged  particle  dynamics 
(relativistic  Lorenz’s  force  law) 
from  first  principles.  Such 
simulations  require  major 
computational  resources  and  are 
scalable  to  hundreds  of  nodes. 


“ICEPIC 

development  has 
been  driven  by  the 
desire  to  achieve 
high  power  in  a 
compact  RF  device.' 


The  team  successfully  applied  ICEPIC  to  simulate  the 
University  of  Michigan’s  relativistic  magnetron  by 
reproducing  the  RF  power  level,  growth  rate,  and 
operating  frequency.  ICEPIC  predicts  300  MW 
(mega-watt)  of  total  extracted  instantaneous  power, 
150  MW  RMS  (Root  Mean  Square),  compared  with 
200  MW  RMS  measured  in  the  experiment.  ICEPIC 
also  predicts  the  experimental  100  nanosecond  (ns) 
delay  in  the  onset  of  RF  after  the  diode  voltage 
reaches  its  peak.  The  operating  frequency  observed 
in  the  ICEPIC  results  agrees  to  within  a  few  percent  of 
the  1 .03  GHz  measured  in  the  experiment.  The 
ICEPIC  results  also  show  that  the  University  of 
Michigan’s  A6  does  not  operate  in  the  desired  pi 
mode  (a  pi  phase  shift  in  the  electric  field  from  cavity 
to  cavity).  In  addition,  the  ICEPIC  results  show  a 
large  upstream  and  downstream  DC  loss  current, 
similar  to  experimental  observation. 
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Figure  1 .  This  shows  the  Titian/PSI  A6 
magnetron  simulated  in  ICEPIC.  This 
snap  shot  shows  the  electric  field  and 
particles  50  ns  after  the  start  of  the  applied 
voltage.  The  iso-surfaces  in  this  figure  are 
the  electric  field  pointing  around  the 
cathode  (theta  direction).  Red  is  in  the 
clockwise  direction  when  facing  upstream 
(left  in  this  graphic)  green  is  in  the  counter 
clockwise  direction.  The  power  extracted 
from  the  magnetron  into  the  wave-guides 
shows  a  pi  phase  advance  between  the 
two  cavities.  The  desired  mode  has  a  2pi 
phase  advance  for  the  extractors,  this 
magnetron  is  operating  in  the  wrong 
mode. 


In  addition  to  simulating  the  University  of  Michigan 
device,  ICEPIC  is  applied  to  the  Titan/PSI  A6 
magnetron  designed  for  AFRL  (Figure  1).  The  results 
verify  the  1-GW  (gigawatt)  capability  of  the  design, 
but  find  severe  mode  competition  that  hampers  the 
axial  extraction  concept  needed  to  meet  the  compact 
size  goal.  The  current  program  provides  for  an 
evolutionary  step  by  extracting  power  radially 
through  two  waveguides.  The  simulations  show 
roughly  1-GW  of  instantaneous  power  extracted  into 
each  arm  (for  a  total  of  1-GW  RMS  power)  when  the 
entire  cathode  region  under  the  anode  block  is 
allowed  to  emit.  However,  there  is  severe  mode 
competition  between  the  pi  and  2pi/3  modes 
(Figure  2).  The  phase  advance  from  cavity  to  cavity 
is  not  constant  in  this  design.  The  phase  advance  for 
the  extracting  cavities  and  the  cavity  between  is  pi/2. 
The  rest  of  the  cavities  have  a  3pi/4  advance,  which 
makes  the  average  phase  advance  2pi/3.  Clearly,  the 
extractors  are  having  a  profound  effect  on  the  overall 
electrodynamics  of  the  device.  This  shows  the 
importance  of  using  3-D  simulation  tools  and 
including  all  the  geometry  of  the  source.  This  issue 


will  not  be  a  problem  for  the 
initial  proof  of  principle  radial 
extraction  experiments,  but 
limits  the  output  power  for  the 
compact  axial  extraction 
concepts  to  about  half  of  the 
required  value.  In  addition, 
the  peak  electric  field  for  these 
modes  is  not  in  the  center  of 
the  anode  block  as  expected, 
but  downstream  of  the  center 
(Figure  3).  Details  of  the  field 
and  particle  structure  inside 
the  magnetron  is  difficult  to 
obtain  from  experiment 
because  of  the  high  electric 
field  stresses  and  electron  bombardment.  Thus,  the 
simulation  results  are  extremely  insightful. 

Developing  Compact,  Multi- Gigawatt 
Microwave  Sources 

The  task  is  to  develop  compact,  multi-gigawatt 
microwave  sources.  The  next  step  is  now  being 
taken  in  AFRL’s  evolution  towards  true  virtual 
prototyping;  the  relativistic  magnetron  is  being 
simulated  before  it  is  built  at  the  lab.  The  details  of 
the  device  that  will  eventually  be  built,  including  the 
geometric  structure  and  the  externally  generated 
magnetic  field  distribution,  will  be  based  on  these 
simulations.  Scientific  visualizations  are  used  for 
three  focuses.  First,  it  is  used  for  communication  of 
complex  physics.  Second,  it  is  useful  for  debugging 
ICEPIC  software  as  well  as  the  input  files.  Lastly,  it 
can  be  used  to  uncover  new  and  unexpected 
emergent  physics. 


Figure  2.  This  shows  the  Titian/PSI  A6  magnetron 
simulated  in  ICEPIC.  This  snap  shot  shows  the  electric 
field  and  particles  50  ns  after  the  start  of  the  applied 
voltage.  The  camera  is  pointing  upstream  toward  the 
pulse  power.  The  iso-surfaces  in  this  figure  are  the 
electric  field  pointing  around  the  cathode  (theta 
direction).  Red  is  in  the  clockwise  direction  when  facing 
upstream,  green  is  in  the  counter  clockwise  direction. 

The  power  extracted  from  the  magnetron  into  the  wave¬ 
guides  shows  a  pi  phase  advance  between  the  two 
cavities.  The  desired  mode  has  a  2pi  phase  advance  for 
the  extractors,  this  magnetron  is  operating  in  the  wrong 
mode.  The  phase  advance  from  cavity  to  cavity  is  not 
constant  in  this  design.  The  phase  advance  for  the 
extracting  cavities  and  the  cavity  between  is  pi/2.  The 
rest  of  the  cavities  have  a  3pi/4  advance,  which  makes 
the  average  phase  advance  2pi/3. 
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Figure  3.  This  shows  the  Titian/PSI  A6  magnetron  simulated  in  ICEPIC.  This  snap  shot  shows  the  electric  field  and  particles  50  ns  after  the  start  of 
the  applied  voltage.  The  peak  electric  field  for  these  modes  is  not  in  the  center  of  the  anode  block  as  expected,  but  downstream  (left)  of  the 
center  (vertical)  yellow  line. 


The  Titan/PSI  A6  magnetron  design  is  improved 
through  extensive  ICEPIC  simulations.  The 
simulations  allow  studies  of  the  consequences  of 
changing  the  vane  depth  and  extraction  slots  to  force 
the  tube  into  the  pi-mode.  Using  simulation  results, 
the  parasitic  mode  is  reduced  in  amplitude  to  ten 
percent  of  the  desired  pi-mode,  and  cathode  end 
caps  are  designed  to  reduce  the  upstream  and 
downstream  loss  current.  The  reduced  loss  current 
increases  the  efficiency  of  the  magnetron.  Plans  are 
to  build  the  improved  design  in  late  2002  and  to 
compare  to  experimental  results  to  simulation  results 
from  ICEPIC. 
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Multiphase  CFD  Simulations  of  Solid  Propellant 
Combustion  in  Modular  Charges  and 
Electrothermal-Chemical  (ETC)  Guns 

Michael  Nusca  and  Paul  Conroy 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 


The  Army  is  exploring  a  variety  of  gun 
propulsion  options  for  indirect  and 
direct-fire  weapons  for  the  legacy 
force  and  the  Future  Combat  System.  As  it 
transforms,  the  Army  has  identified  requirements  for 
hypervelocity  projectile  launch  for  strategic  Army 
missions.  Among  these  gun  systems  are  those  that 
use  solid  propellant  —  granular  form  loaded 
in  modules  (indirect-fire)  or  disk/strip  form 
for  high-loading-density  cartridges  (direct- 
fire)  —  along  with  electrothermal-chemical 
(ETC)  augmentation.  Advanced  indirect-fire 
gun  systems  with  complex  grain  and 
container  geometries  are  being  investigated 
for  the  Modular  Artillery  Charge  System 
(MACS)  that  has  been  developed  for  all 
fielded  155  mm  cannons.  The  gun 
propulsion-modeling  environment  has  been 
one  in  which  separate  codes  (some  1-D, 
some  2-D)  are  used,  with  no  single  multi¬ 
dimensional  code  able  to  address  the  truly 
3-D  details  of  all  of  these  systems.  This 
situation  renders  comparison  of  ballistic 
performance  cumbersome  and 
inconclusive.  In  contrast,  the  multiphase 
continuum  equations  that  represent  the 
physics  of  gun  propulsion  comprise  a  set  of  ™ 
general  equations  applicable  to  all  of  these 
systems.  Thermodynamic  state  equations  and 
constitutive  laws,  as  well  as  boundary  conditions, 
represent  the  relevant  differences  in  system  details. 

NGEN  -  The  Army’s  Next  Generation 
Interior  Ballistics  Model 

The  Army’s  next  generation  interior  ballistics  model, 
NGEN,  is  a  multi-dimensional,  multi-phase  CFD  code 
that  has  been  under  development  at  ARL  for  a 
number  of  years.  The  NGEN  code  incorporates  3-D 


“The  success  of  this 
modeling  effort  has 
yielded  a  code  that 
stands  as  a  singular 
computational 
simulation  tool 
providing  the  Army 
with  the  unique 
capability  to  simulate 
current  and  emerging 
gun  propulsion 
systems.  These 
simulations  serve  to 
both  streamline  testing 
and  aid  in  the 
optimization  of  interior 
ballistics 
performance." 


continuum  equations  along  with  auxiliary  relations 
into  a  single  code  structure  that  is  both  modular  and 
readily  transportable  between  scaleable,  high 
performance  computer  architectures.  Since  interior 
ballistics  involves  flowfield  components  of  both  a 
continuous  and  a  discrete  nature,  an  Eulerian/ 
Lagrangian  approach  is  used.  The  components  of 
the  flow  are  represented  by  the  balance 
equations  for  a  multi-component  reacting 
mixture.  This  describes  the  conservation  of 
mass,  momentum,  and  energy  on  a 
sufficiently  small  scale  of  resolution  in  both 
space  and  time.  A  macroscopic 
representation  of  the  flow  is  adopted  using 
these  equations  derived  by  a  formal 
averaging  technique  applied  to  the 
microscopic  flow.  These  equations  require  a 
number  of  constitutive  laws  for  closure 
including  molecular  transport  terms,  state 
equations,  chemical  reaction  rates, 
intergranular  stresses,  and  interphase 
transfer.  The  numerical  representation  of 
these  equations  as  well  as  the  numerical 
solution  is  based  on  a  finite-volume 
discretization  and  high-order  accurate, 
conservative  numerical  solution  schemes. 


The  success  of  this  modeling  effort  has 
yielded  a  code  that  stands  as  a  singular 
computational  simulation  tool  providing  the  Army 
with  the  unique  capability  to  simulate  current  and 
emerging  gun  propulsion  systems.  These 
simulations  serve  to  both  streamline  testing  and  aid 
in  the  optimization  of  interior  ballistics  performance. 
High  performance  computing  resources  and 
scientific  visualization  tools  are  essential  to  efficiently 
use  these  numerical  simulations,  because  the  NGEN 
code  solves  the  governing  equations  in  a  high- 
resolution  2-D  and  3-D  mode.  In  one  particularly 
significant  application,  the  NGEN  code  was  used  to 


-123- 


-  Deny  Enemies  Sanctuary 


Figure  1 .  Scientific  visualization  of  the  MACS  charge  (five 
modules)  showing  the  location  of  propellant  particles  (colored  by 
temperature)  and  formation  of  gas  dynamic  vortices  in  the  gun 
chamber 


simulate  the  MACS  propelling  charge,  that  is  being 
developed  at  the  Armament  Research  Development 
and  Engineering  Center  for  all  fielded  155mm 
cannon,  and  for  the  XM297  Crusader  cannon 
(developed  by  PM-Crusader).  Figure  1  displays  a 
scientific  visualization  of  the  MACS  using  NGEN.  The 
NGEN  code  simulations  for  the  M231 ,  low-zone 
MACS  charge,  aided  in  the  successful  type 
classification  of  the  charge.  For  the  high-zone 
charge,  NGEN  code  simulations  were  used  to 
uncover  previously  unknown  physical  phenomena  for 
the  M232  charge  that  only  occurs  during  charge 


fallback  in  an  elevated  gun  tube.  This  is  a  particularly 
acute  concern  at  top  zone  since  bag  charges 
positioned  adjacent  to  the  breech  lead  to  large- 
amplitude  pressure  waves  that  can  destroy  the 
weapon.  NGEN  simulations  of  these  modular 
charges  proved  that  the  prevailing  gas  dynamic 
process  (when  the  charges  are  positioned  adjacent 
to  the  breech  face)  causes  the  modules  to  be 
displaced  forward.  This  mitigates  pressure  wave 
formation  —  a  fact  confirmed  by  actual  gun  tests 
performed  a  year  later. 

Encouraged  by  these  successes,  the  scientists  at 
ARL  used  the  NGEN  code  to  investigate  the  loading 
of  M232  charges  that  are  at  the  ambient  temperature 
along  with  modules  that  have  been  initially 
temperature  conditioned.  This  combination  emulates 
taking  modules  from  different  loading  pallets  and 
assembling  a  single  propelling  charge.  The 
important  questions  were  whether  such  a  charge 
would  lead  to  high-amplitude  pressure  waves  in  the 
gun  chamber  and  what  level  of  performance  gain  or 
loss  could  be  expected.  Figure  2  shows  a  scientific 
visualization  of  the  case  of  three  hot  modules  loaded 
in  the  forward  position  along  with  two  ambient 
modules  comprising  the  final  charge.  In  this  situation 
(considered  a  worst-case  scenario)  hot  propellant 
impacts  the  projectile  base,  as  predicted  by  the 
NGEN  code.  This  leads  to  a  noticeable  amplification 
of  pressure  waves  in  the  chamber  but  still  non¬ 
threatening  to  the  weapon’s  structural  integrity.  The 
predicted  projectile  velocity  was  shown  to  be  slightly 
enhanced  by  this  type  of  charge. 
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Figure  2.  Scientific  Visualization  of  the  MACS  charge  (five  modules  with  three  forward  modules  at  initially  elevated  temperature)  showing  the 
location  of  propellant  modules  (colored  by  temperature)  at  eight  times  in  the  flamespreading  process.  Note  the  impact  of  hot  propellant  on  the 
projectile  base  (E,F). 
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Direct-Fire  Gun  Simulation 

In  another  application,  this  time  to  direct-fire  gun 
systems,  the  ARL  researchers  used  the  NGEN  code 
to  simulate  solid  propellant  charges  ignited  using  an 
ETC  igniter.  A  simulation  of  a  high-loading-density 
(HLD)  M829A1  120mm  tank  charge  composed  of 
disk  and  granular  solid  propellant  media 
demonstrated  the  ability  of  the  NGEN  code  to 
accommodate  ignition  and  flamespreading  through  a 
novel  charge.  This  simulation  included  the  effects  of 
projectile  afterbody  intrusion  and  movement  as  well 
as  compression  and  the  resulting  ignition  delay  in  the 
propellant.  Figure  3  is  a  scientific  visualization  of  this 
gun  charge.  These  simulations  revealed,  for  the  first 
time,  that  the  igniter,  positioned  at  the  gun  breech, 
could  forcefully  compress  the  disk  charge.  This 
compression  closed  critical  flow  paths  in  the  charge, 
preventing  effective  flamespreading  and  even 
ignition.  The  simulations  showed  that  the  low 
molecular  weight  plasma,  generated  by  an  ETC 
igniter,  could  circumvent  this  problem.  The  plasma 
establishes  a  convectively  driven  flame  that 
propagates  faster  than  the  material  compression 
wave  in  the  disk  propellant,  permitting  an  even 
ignition  of  the  charge.  A  basic  design  tenet  for  using 
high-loading-density  charges  was  established;  that 
pressure  waves  in  the  gun  chamber  generated  by  the 
conventional  igniter,  when  paired  with  the  disk 
propellant  charge,  were  avoided  by  using  an  ETC 
igniter. 
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Figure  3.  Scientific  Visualization  of  a  HLD  charge  (disk  propellant)  showing 
gas  pressure  (colored  by  pressure  up  to  red  at  about  100  MPa).  Note  the 
higher  pressure  forward  on  the  projectile  afterbody. 
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The  Effects  of  Propellant  Deformation  During  a 
Gun  Firing 

Stephen  Ray 

Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 


The  US  Army  is  studying  the  use  of 

increased  amounts  of  propellant  in  tank 
and  artillery  gun  rounds  to  increase  the 
range  and  lethality  of  the  guns.  One  of  the 
technical  challenges  posed  by  larger  amounts  of 
propellant  is  ensuring  that  all  of  the  propellant  burns 
rapidly  enough  while  keeping  the  pressure  in  the  gun 
below  certain  levels.  Usually,  higher  pressure  is  an 
effect  of  faster  propellant  burning.  If  the  projectile 
leaves  the  gun  before  the  propellant  has  finished 
burning,  the  energy  released  by  combustion  after  the 
projectile  exits  is  wasted.  On  the  other  hand,  if  the 
pressure  in  the  gun  is  too  high,  the  gun  can  be 
damaged. 

Army  researchers  want  better  control  of  the  tradeoff 
between  the  speed  at  which  the  propellant  burns  and 
the  pressure  in  the  gun.  In  order  to  achieve  that 
control,  they  first  need  a  better  understanding  of 
some  of  the  detailed  phenomena  that  occur  while  the 
gun  is  being  fired.  For  instance,  propellant  disks  can 
be  placed  in  the  gun  (Figure  1),  and  they  start  to  burn 
because  hot  gas  from  the  igniter  reaches  them  and 
raises  their  surface  temperature.  The  hot  gas  will 
also  cause  the  propellant  disks  to  move,  deform,  and 
possibly  break.  This  interaction  between  the  gas  and 
the  propellant  is  not  yet  well  understood,  but  knowing 
how  it  works  will  help  researchers  control  the  burning 
of  the  propellant. 


iliijiiiiinniiiiiiiiiiiiiiiiiijjJ 

iiiiiimiimimiiiimi' 


i ' . 


I'Tun.illLt 


PinpCl[»l  Dlda 


i  Ik  I'm  iu ■ 


Figure  1 .  Sketch  of  disks  of  solid  propellant  in  a  gun  round 


Numerical  Study  of  Gas  and 
Propellant  Interaction  Underway 

Studying  the  interaction  between  the  gas  and 
propellant  is  difficult  to  do  experimentally.  An  effort  to 
study  this  numerically  is  underway  using  HPC 
resources  available  at  AHPCRC.  In  order  to  study 
this  problem  in  detail,  the  motion,  pressure,  and 
temperature  of  the  gas  have 
to  be  computed  in  each 
time  step  using  conservation 
of  mass,  momentum,  and 
energy  equations.  The 
motion,  stresses,  and  strains 
of  the  propellant  are 
computed  by  using  the 
equations  of  motion 
complemented  by  an 
appropriate  strength  model 
and  an  equation  of  state. 

This  all  has  to  be  done  on  a 
very  fine  grid,  with  the 
distance  between  grid 
points  on  the  order  of  0.008 
inches.  The  HPC  systems  have  the  memory  and 
computing  speed  needed  to  study  this  problem. 

Scientific  visualization,  specifically  the  FV2  and 
PRESTO  visualizers  developed  at  the  AHPCRC,  has 
played  an  important  role  in  understanding  the  data 
generated  by  the  numerical  model.  In  the  study  of 
gas  interacting  with  the  propellant,  the  formation  and 
motion  of  pressure  waves  in  the  gas  is  a  very 
important  phenomenon.  Creating  animations  of  the 
pressure  in  the  gas  allows  researchers  to  see  how  it 
happens,  which  aids  in  understanding  it. 


“Army  researchers  want 
better  control  of  the 
tradeoff  between  the 
speed  at  which  the 
propellant  burns  and  the 
pressure  in  the  gun.  This 
interaction  between  the 
gas  and  the  propellant  is 
not  yet  well  understood, 
but  knowing  how  it  works 
will  help  researchers 
control  the  burning  of  the 
propellant.” 
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Numerical  Study  Gives  New  Insight 

The  numerical  study  of  the  interaction  between  the 
gas  and  the  propellant  on  the  HPC  systems  has 
given  new  insights  into  some  of  the  effects  of  the 
interaction.  For  instance,  many  researchers  agree 
that  the  propellant  disks  will  break  if  they  are  involved 
in  a  high-speed  collision.  But  the  HPC  study  shows 
that  the  gas  can  bend  the  disks  around  the 
neighboring  propellant  and  break  them  even  though 
the  disks  are  moving  at  very  low  speeds  (Figure  2). 
The  inclusion  of  combustion  and  fracture  models  in 
the  simulation  of  the  propellant  will  allow  the  code  to 
generate  new  details  about  the  interactions.  Such 
insights  will  give  researchers  more  information  as 
they  develop  a  fuller  understanding  of  the  gun  firing, 
which  in  turn  will  develop  the  control  needed  to 
safely  increase  the  amount  of  propellant  in  the  tank 
and  artillery  gun  rounds. 


References 

S.  Ray,  “The  Modeling  of  Combustion  in  a  Solid 
Propellant  Gun”,  presented  at  the  38th  Joint  Army- 
Navy-NASA-Air  Force  (JANNAF)  Combustion 
Subcommittee  Meeting,  sponsored  by  the  Chemical 
Propulsion  Information  Agency,  Destin,  FL,  April 
2002. 

S.  Ray,  “A  Study  of  Fluid-Structure  Interaction 
Phenomena  in  Interior  Ballistics  Applications”, 
presented  at  the  31st  American  Institute  of 
Aeronautics  and  Astronautics  (AIAA)  Fluid  Dynamics 
Conference,  sponsored  by  the  AIAA,  Anaheim,  CA, 
June  2001 . 

CTA:  Computational  Fluid  Dynamics 
Computer  Resources:  Cray  T3E  [AHPCRC] 


Figure  2.  Gas  temperature  in  barrel  end  of  the  gun.  Notice  how  one 
disk  is  bending  over  its  neighbor;  this  disk  is  fracturing. 
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Modeling  and  Simulation  of  Nanoparticle 
Formation  and  Growth  in  Turbulent  Reacting  Flows 

Sean  Garrick 

Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 


Simulations  can  be  used  in  the  production 

and  characterization  of  nanoscale  energetic 
particles  and  composites.  There  are 
several  technologies  that  can  be  employed 
in  the  manufacture  of  nanoscale  materials.  Vapor- 
phase  methodologies  are  by  far  the  most  favored 
because  of  the  chemical  purity  and  cost 
considerations.  When  driven  from  gas  phase 
precursors,  there  are  several  phenomena,  which 
need  to  be  addressed:  vapor-phase  chemistry, 
particle  formation,  and  growth  (coagulation, 
coalescence,  condensation,  etc.)  among  others. 

Researchers  at  AHPCRC  are  developing  tools  to 
simulate  the  production  of  nanoenergetic  particles  — 
propellants  or  explosives.  Energetic  nanoparticles 
have  been  targeted  as  one  of  the  propellants  for  the 
Army’s  FCS  because  they  exhibit  very  high  reaction 
rates.  However,  the  Army  currently  has  no  source  of 
these  materials.  The  manufacture  of  nanoparticles  is 
difficult  because  all  of  the  relevant  mechanisms/ 
dynamics  are  not  completely  understood.  To 
mitigate  this,  tools  are  being  developed  to  determine 
the  effects  of  chemistry,  flow/temperature  dynamics, 
and  particle-particle  interactions  on  nanoparticle 
formation  and  growth  in  realistic  flow  systems,  such 
as  those  employed  in  cost-effective  processes  like 
vapor-phase  synthesis.  In  this  way,  the  Army  can 
have  particles  in  a  specified  size  range,  chemical 
composition,  and  physical  structure. 

Simulating  Nanoscale  Particle 
Production 

The  numerical  simulation  of  nanoscale  particle 
production  through  gas  phase  synthesis  involves  the 
solution  of  a  large  number  of  coupled,  non-linear 
integro  and  partial  differential  equations.  These 
models  represent  turbulent  reacting  flows  with  length 
scales  spanning  seven  orders  of  magnitude.  With 
the  addition  of  nucleation,  coagulation,  and  other 


particle-particle  interactions,  a  rather  diverse  range  of 
phenomena  is  under  investigation.  The  primary 
objective  is  to  improve  our  physical  understanding  of 
particle  formation  and  growth  in  a  turbulent  reacting 
environment. 

To  illustrate  the 
underlying  phenomena, 
simulations  of  several 
turbulent  flows  were 
performed.  A  turbulent 
planar  jet  was  simulated 
with  a  domain 
comprised  of  24,500,000 
grid  points.  In  this 
calculation,  the  time 
steps  are  on  the  order  of 
1 0'9  seconds  and  the 
dynamics  of  particles 
between  1  and  10 
nanometers  were 
captured.  The 
concentration  of  4  nanometer  diameter  particles  is 
shown  in  Figure  1  (a).  From  this  image,  one  can  see 
that  the  particles  are  engulfed  in  the  vortical 
structures  and  are  effectively  segregated  from  the 
rest  of  the  flow.  The  characteristic  flow  time  of  the 
particles,  the  length  of  time  the  particles  remain  in 
specific  regions  of  the  flow,  and  the  temperature  of 
these  regions  are  key  areas  of  investigation.  These 
are  just  a  few  of  the  issues  which  can  be  answered 
with  the  capability  facilitated  by  advanced  simulation. 

High  performance  computing  resources  were 
required  to  run  the  direct  numerical  simulation  of  the 
turbulent  jet.  The  exact  solution  required  96,265  CPU 
hours.  Other  solution  methodologies  had  to  be 
explored  for  the  more  complex  flow  geometries. 
Large-eddy  simulations  can  provide  reasonably 
accurate  simulations  in  a  fraction  of  the  time  required 
by  direct  numerical  simulation.  However,  the  effect  of 
turbulence  on  both  the  flow  and  particle  fields  must 
be  modeled. 


“Energetic  nanoparticles  have 
been  targeted  as  one  of  the 
propellants  for  the  Army’s  FCS 
because  they  exhibit  very  high 
reaction  rates.  Tools  are  being 
developed  to  determine  the 
effects  of  chemistry,  flow/ 
temperature  dynamics,  and 
particle-particle  interactions  on 
nanoparticle  formation  and 
growth  in  realistic  flow 
systems,  such  as  those 
employed  in  cost-effective 
processes  like  vapor-phase 
synthesis.” 
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Large-Eddy  Simulation  versus  Direct 
Numerical  Simulation 

To  test  the  methodology,  both  large-eddy  simulations 
and  direct  numerical  simulations  of  nanoparticle 
coagulation  in  temporal  mixing  layers,  which  are 
subsections  of  the  turbulent  planar  jet  were 
performed.  The  results  are  shown  in  Figure  2.  The 
concentration  of  4  nanometer  diameter  particles 
obtained  through  direct  numerical  simulation  is 
shown  in  Figure  2(a)  and  the  concentration  obtained 
through  large-eddy  simulation  is  shown  in  Figure 
2(b).  The  large-eddy  simulation  yields  very  good 
results  while  reducing  the  CPU  time  by  a  factor  of 
more  than  400.  The  use  of  large-eddy  simulation  will 
greatly  affect  the  development  of  advanced 
nanostructured  materials  such  as  composites,  which 
are  being  considered  as  the  next  generation  of 
propellants  because  of  the  large  surface-to-volume 
ratio,  and  thus  are  highly  reactive. 


Figure  1 .  Nanoparticle  coagulation  in  a  turbulent  jet:  (a) 
Concentration  of  4  nm  diameter  particles;  (b)  Turbulent  growth 
rate 
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Computer  Resources:  Cray  T3E  [AHPCRC] 
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Figure  2.  Nanoparticle  growth  in  temporal  mixing  layer:  (a)  Direct 
numerical  simulation  (450  CPU  hours);  (b)  Large-eddy  simulation  (1 
CPU  hour) 


The  effect  of  turbulence  on  the  particle  growth  rate  is 
shown  in  Figure  1  (b).  The  figure  reveals  that  the 
effect  is  more  pronounced  as  the  flow  develops  and 
is  as  high  as  20%.  This  resulted  in  the  development 
of  the  first  large-eddy  simulation  of  nanoparticle 
growth  in  turbulent  flows,  which  represents  the 
particle  field  in  a  manner  which  allows  for  the  capture 
of  the  underlying  physics  and  chemistry  of  particle 
coagulation. 
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3.21  angstroms  indicates  that  this  cation  is  unstable  with  respect  to  dissociation  to  triphenylmethyl  cation  and  N2. 
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The  overarching  strategic  goal  for  the  DoD  is  to  maintain  US  global  superiority  in  space  while  making 
space  access  and  operations  easily  affordable.  From  space,  a  whole  range  of  critical  information 
collection  and  distribution  functions  becomes  possible  with  both  robust  global  reach  and  little  forward- 
based  infrastructure.  Information  provided  to  US  military  personnel  by  space-based  systems  includes 
weather,  forces  location/movement,  environmental  monitoring,  transportation  routes,  advanced 
warning  on  weapons  deployment,  and  weapons  targeting.  Future  space  systems  could  allow 
application  of  space-based  force  against  ballistic  missiles  and  other  threats.  The  number  of  space- 
faring  nations  and  their  capabilities  is  increasing  and,  with  that,  the  possibility  of  threat  to  US  forces. 
Space  systems  based  on  current  technology  are  highly  expensive  to  acquire  and  operate.  This  high 
cost  threatens  US  dominance  of  space. 

One  of  the  ways  to  reduce  the  costs  of  future  space  systems  is  to  reduce  the  weight  of  the  vehicle.  A 
common  area  for  advancement  to  reduce  the  weight  of  launch  vehicles  is  in  the  propellants. 

Propellants  comprise  70-90%  of  a  launch  vehicle’s  weight  and  accounts  for  40-60%  of  the  cost  of  the 
vehicle.  Added  to  this,  the  manpower  required  to  load  the  propellants  into  the  space  vehicle  and  the 
costly  safety  procedures  and  apparel  that  are  intrinsic  in  this,  research  in  more  efficient  and  safer 
propellants  is  ongoing. 

Another  focus  is  the  development  of  rocket  propulsion  engines  and  motors  with  improved  performance 
for  transition  into  existing  or  new  systems.  Boost  and  orbit  transfer  propulsion  systems  will 
demonstrate  improvements  in  specific  impulse,  mass  fraction,  thrust  to  weight,  reliability,  reusability, 
and  cost.  Spacecraft  propulsion  systems  will  demonstrate  improvements  in  specific  impulse,  thruster 
efficiency,  and  mass  fraction. 

Both  of  the  above  areas  have  one  commonality  —  designing  the  item  prior  to  developing  the  item. 

High  performance  computing  is  the  key  to  these  design  efforts.  The  articles  by  Boatz,  Sorescu,  and 
Thompson;  by  McQuaid;  and  another  article  by  Boatz  describe  quantum  mechanical  calculations  to 
determine  which  proposed  propellants  will  provide  sufficient  energy  as  a  propellant,  if  there  is  a 
feasible  path  to  produce  it,  and  if  it  is  less  toxic.  All  three  of  these  characteristics  must  be  considered 
prior  to  the  development  of  new  propellants.  In  the  article  by  Gaitonde,  the  design  of  facets  of 
supersonic  engines  is  studied  to  alleviate  problems  with  sustained  hypersonic  flight. 

We  believe  that  using  HPC  in  the  early  stages  of  developing  new  space  technologies  helps  ensure 
continued  US  superiority  in  space. 


Dr,  Leslie  Perkins 

High  Performance  Computing  Modernization  Program  Office,  Arlington,  VA 
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Simulations  of  Energetic  Materials  for  Rocket 
Propulsion:  Getting  More  Bang  for  the  Buck 

Jerry  Boatz 

Air  Force  Research  Laboratory  (AFRL),  Edwards  Air  Force  Base,  CA 
Dan  C.  Sorescu 

University  of  Pittsburgh,  Pittsburgh,  PA 

Donald  L,  Thompson 

Oklahoma  State  University,  Stillwater,  OK 


Powdered  aluminum  has  long  been  used 
as  an  energetic  ingredient  in  rocket 
propellant  formulations,  comprising 
approximately  15-20%  of  some 
conventional  ammonium  perchlorate  solid 
propellants.  However,  the  performance  of  aluminum 
is  reduced  because  of  the  rapid  formation  of  an 
aluminum  oxide  overcoat  on  aluminum  particles 
before  combustion,  which  inhibits  efficient  burning. 
Furthermore,  formation  of  the  oxide  overcoat  severely 
reduces  the  potential  advantages  of  using  high 
surface-to-volume-ratio  ultrafine  aluminum  particles, 
which  have  highly  desirable  properties  such  as 
enhanced  burn  rates. 

Inhibiting  Formation  of  Oxide 
Overcoat 

In  order  to  inhibit  the  rapid  formation  of  an  oxide 
overcoat  on  the  ultrafine  aluminum  particles  without 
simultaneously  degrading  performance,  coating  the 
aluminum  particles  with  an  energetic  material  such 
as  1 ,3,5,7-tetranitro-1 ,3,5,7-tetraazacyclooctane 
(HMX)  has  been  proposed.  One  of  the  objectives  of 
this  challenge  project  is  to  characterize  the 
interactions  between  energetic  compounds  such  as 
HMX  and  aluminum  surfaces  in  order  to  understand, 
at  an  atomic  level,  the  chemistry  at  the  metal- 
molecule  interface.  Specifically,  ab  initio  quantum 
chemical  calculations  are  used  to  determine  if  HMX 
and/or  other  classes  of  energetic  materials  will 
effectively  inhibit  the  formation  of  an  oxide  overcoat 
on  the  surface  of  aluminum. 


Using  Quantum  Chemical 
Calculations 

Conventional  techniques  for 
determining  the  feasibility  of 
new  ideas  such  as  this  have 
relied  heavily  on  costly  and 
time-consuming  experimental 
synthesis  and  testing. 

However,  the  availability  of 
HPC  resources  has 
significantly  lessened  the  need 
for  these  empirical  approaches 
by  enabling  the  application  of 
reliable  high-level  quantum 
chemical  calculations  to  address  complex  issues 
such  as  the  nature  of  the  interactions  between 
energetic  materials  and  metallic  surfaces.  A  key 
advantage  of  using  HPC  is  the  ability  to  efficiently 
screen  a  variety  of  energetic  materials  as  potential 
metallic  coatings  and  focus  subsequent 
experimental  efforts  on  only  the  most  promising 
candidates. 

Spin-polarized  generalized  gradient  approximation 
(GGA)  density  functional  theory,  using  ultrasoft 
Vanderbilt-type  pseudopotentials  and  the  PW91 
exchange-correlation  functional  in  a  plane-wave 
basis,  has  been  used  to  characterize  the  interactions 
between  the  energetic  molecules  nitromethane  (NM), 
HMX,  and  1 ,1 -diamino-2, 2-dinitroethylene  (FOX-7) 
and  the  aluminum  (111)  surface  (Figure  1).  Two 
different  models  have  been  used  to  approximate  the 

(1 1 1)  surface  of  aluminum:  a  (V7xV7)R19.1°  slab 


"A  key  advantage  of 
using  HPC  is  the  ability 
to  efficiently  screen  a 
variety  of  energetic 
materials  as  potential 
metallic  coatings  and 
focus  subsequent 
experimental  efforts  on 
only  the  most 
promising  candidates." 
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Figure  1 .  Initial  (left)  and  final  (right)  configurations  of  an  HMX 
molecule  on  an  aluminum  (111)  surface.  In  the  final  configuration, 
one  of  the  oxygen  atoms  is  bonded  to  two  surface  aluminum  atoms 
and  is  completely  dissociated  from  the  initial  N02  group.  The  other 
oxygen  atom  also  forms  a  bond  with  aluminum  but  remains  bonded 
to  the  nitrogen  atom  in  the  original  N02  group. 


surface  will  be  computed.  Researchers  will  attempt 
to  understand  the  type  of  chemical  processes  that 
can  take  place  at  the  interface  of  aluminum  hydrides 
with  different  classes  of  energetic  materials.  In  this 
case,  they  will  focus  on  the  case  of  ammonium  nitrate 
and  RDX  crystals  where  the  detonation  properties  are 
significantly  affected  by  the  interaction  with  aluminum 
hydrides. 

References 
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model  with  four  layers  (28  aluminum  atoms)  used  for 
NM  and  a  four-layer  (3x3)  slab  (36  atoms)  for  the 
larger  molecules  (HMX  and  FOX-7).  All  calculations 
were  performed  using  the  Vienna  as  Initio  Simulation 
Package  (VASP)  code. 

Preliminary  predictions  of  the  interactions  between 
the  energetic  molecules  NM,  HMX,  and  FOX-7  have 
indicated  a  common  propensity  for  oxidation  of  the 
aluminum  surface  by  the  oxygen-rich  nitro  groups 
(N02)  present  in  these  molecules.  Dissociation  of  the 
nitro  groups  and  formation  of  strong  AI-0  bonds 
appears  to  be  a  common  mechanism  for  these 
molecules.  These  results  suggest  that  nitro- 
containing  energetic  compounds  are  not  likely  to  be 
effective  coating  materials  for  preventing  rapid 
oxidation  of  aluminum. 

Scientific  visualization  is  used  for  the  construction  of 
the  initial  atomic  configurations  for  crystals,  surfaces 
and  gas-solid  systems,  using  the  Crystal  and  Surface 
Builders  modules  available  in  Cerius2  package. 

Interactions  of  Several  Ionic  Systems 

Future  work  will  extend  this  set  of  investigations  to 
first  principles  density  functional  theory  calculations 
of  the  interactions  of  several  ionic  systems  such  as 
ammonium  nitrate  or  ammonium  dinitramide  with  an 
Al  surface.  In  these  investigations,  key  reaction 
pathways  of  these  energetic  salts  on  an  aluminum 
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A  New  Design  Approach  to  Hydrazine-Alternative 
Hypergols  for  Missile  Propulsion 

Michael  McOuaid 

Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 


Providing  an  extremely  reliable  basis  on 

which  to  design  intermittent  and/or  variable 
thrust  propulsion  systems,  hypergolic  liquid 
(or  gel)  fuel-oxidizer  combinations — i.e., 
those  that  react  spontaneously  upon  mixing — are 
candidates  for  propelling  beyond  line-of-sight 
weapons  for  the  Army’s  FCS,  an  example  being  the 
Loitering  Attack  Missile  being  developed  under  the 
NetFires  technology  demonstration  program. 

Finding  Viable  Alternatives  to  Toxic 
and  Carcenogenic  Fuels 

One  of  the  drawbacks  to  “standard”  hypergolic  fuel- 
oxidizer  combinations  is  that  the  fuels  are  derived 
from  hydrazine,  monomethylhydrazine  (MMFI),  and/ 
or  unsymmetrical  dimethylhydrazine  (UDMH) — all  of 
which  are  acutely  toxic  and  suspected  carcinogens. 
Fielding  weapons  systems  with  them  necessitates  the 
implementation  of  burdensome  and  costly  handling 
procedures.  Moreover,  the  only  US  producer  of 
hydrazines  is  in  danger  of  closing.  Searching  for 
alternatives  to  hydrazine-based  fuels,  the  Army,  in 
collaboration  with  the  Air  Force  and  the  Navy,  is 
investigating  the  use  of  secondary  and  tertiary  amine 
azides  in  various  propulsion  applications.  One  of  the 
formulations  within  this  class  of  compounds  receiving 
considerable  attention  is  2-azido-N,N- 
dimethylethanamine  [(CFI3)2NCFI2CFI2N3].  Also 
referred  to  as  DMAZ,  this  fuel  has  shown  great 
promise  as  a  hypergolic  fuel,  performing 
competitively  with  Aerozine  50  (a  50/50  mixture  of 
hydrazine  and  UDMH)  in  inhibited  red  fuming  nitric 
acid  (IRFNA)  oxidized  systems.  DMAZ-IRFNA 
systems  do  not,  however,  meet  “ignition  delay” 
standards  set  by  MMH-IRFNA  systems.  This 
shortcoming  negatively  impacts  engine  design  and 
may  preclude  the  fielding  of  DMAZ. 


Gaining  Insights  Through 
Computational  Chemistry  in  a  High 
Performance  Computing 
Environment 

Based  on  traditional  chemical  insights,  proton 
transfer  to  the  amine  nitrogen  has  been  hypothesized 
as  an  important  initial  step  in  the  ignition  of 
DMAZ-IRFNA  systems.  Therefore,  attempts  to 
address  the  ignition  delay  issue  have  been  based  on 
synthesizing  2-azidoethanamines  with  amine 
substituents  other  than  the 
methyl  groups  found  in  DMAZ, 
with  the  hope  being  to  increase 
the  nucleophilic  nature  of  the 
amine  lone  pair  electrons. 

However,  a  computational 
quantum  chemistry  study  of 
DMAZ  identified  that  in  its 
lowest  energy  configuration 
(Figure  1),  the  amine  nitrogen  is 
shielded  from  proton  attack  by 
the  azide  group,  and  therefore 
may  be  a  barrier  to  the  ignition 
process.  Moreover,  it  is  not 
possible  to  prevent  azide  group 
shielding  of  the  amine  lone  pair 
in  alternately  substituted  2-azidoethanamines.  The 
finding  stimulated  a  search  for  molecular  designs  that 
prevented  shielding,  but  did  not  otherwise 
significantly  compromise  DMAZ’s  features  and 
thermophysical  properties.  Proposing  to  achieve  this 
goal  by  introducing  a  cyclic  structure  between  the 
amine  and  azide  groups,  ballistic  performance¬ 
relevant  properties  of  the  c/s  and  trans  isomers  of  a 
notional  compound — 2-azido-N,N- 
dimethylcyclopropanamine  (ADMCPA) — were 
computationally  evaluated.  (The  trans  form  of 
ADMCPA  prevents  direct  amine-azide  interaction). 


"The  ability  to 
computationally  screen 
the  properties  and 
predict  the 
performance  of 
notional  hypergols 
through  this 
environment  has 
provided  firm 
justification  for  further 
investment  in  this  high- 
risk  development 
effort.” 
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Figure  1 .  Molecular  structure  of  the  lowest  energy  DMAZ  structure  (left)  and  its  electrostatic  potential  on  an  electron  isodensity 
surface  (right).  Dark  blue  regions  of  surface  indicate  sites  to  which  nitric  acid  will  transfer  a  proton,  the  amine  site  being  screened  by 
the  azide  group. 


Molecular  stability  and  gas-phase  heats  of  formation 
were  established  through  computational  quantum 
chemistry,  and  liquid  densities  and  heats  of 
vaporization  were  estimated  via  molecular  dynamics 
simulations.  The  results  were  then  used  as  input  to 
thermochemical  calculations  of  ballistic  potential. 
Finding  that  the  ADMCPA  isomers  merit  further 
investigation  as  hypergols,  their  synthesis  is  now 
being  pursued  by  the  Army  Aviation  and  Missile 
Research,  Development,  and  Engineering  Center  as 
an  option  for  its  Impinging  Stream  Vortex  Engine — a 
top  launch  package  candidate  for  the  Loitering  Attack 
Missile. 

Without  the  insight  provided  by  computational 
quantum  chemistry  in  a  high  performance  computing 
environment,  the  new  design  approach  could  not 


have  been  identified.  Moreover,  the  ability  to 
computationally  screen  the  properties  and  predict  the 
performance  of  notional  hypergols  through  this 
environment  has  provided  firm  justification  for  further 
investment  in  this  high-risk  development  effort. 
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New  Polynitrogen  Molecules  -  Energetic  "Air”  as  a 
Next-Generation  Propellant? 

Jerry  Boatz 

Air  Force  Research  Laboratory  (AFRL),  Edwards  Air  Force  Base,  CA 


The  identification,  development,  and 

formulation  of  new  energetic  materials  for 
advanced  rocket  propulsion  applications 
are  areas  of  long  standing  interest  to  the  Air 
Force.  The  performance  limits  of  currently  used 
propellants  have  been  reached,  so  new  energetic 
compounds  are  required  to  significantly  improve  the 
ability  of  the  warfighter  to  access  and  control  space. 

Polynitrogen  species  such  as  the  recently  discovered 
N5+  cation  (a  positively  charged  ion)  are  of  interest 
as  potential  energetic  ingredients  in  new  propellant 
formulations.  The  recent 
successful  synthesis  of  N5+  in 
macroscopic  quantities  has 
prompted  the  search  for 
additional  polynitrogen 
compounds.  Computational 
chemistry  plays  a  central  role 
in  determining  the  stabilities, 
potential  synthetic  pathways, 
and  key  spectroscopic 
“fingerprints”  of  new 
polynitrogen  species. 

Characterizing  New  Materials 

Conventional  techniques  for  characterizing  new 
materials  such  as  polynitrogens  have  relied  heavily 
on  costly  and  time-consuming  experimental 
synthesis  and  characterization.  However,  the 
availability  of  HPC  resources  has  significantly 
lessened  the  need  for  these  more  empirical 
approaches  by  enabling  the  application  of  high-level 
quantum  chemical  calculations  to  reliably  predict 
important  properties  such  as  heats  of  formation, 
stabilities,  mechanisms  of  formation  and 
decomposition,  and  spectroscopic  constants.  A  key 
advantage  of  using  HPC  in  this  regard  is  the  ability  to 
efficiently  “screen”  a  large  number  of  potential 
polynitrogen  molecules  and  focus  subsequent 


experimental  efforts  on  the  subset  of  only  the 
promising  candidates. 

Computing  Chemical  Characteristics 
Using  Ab  Initio  Methods 

The  structures,  stabilities,  vibrational  frequencies, 
and  infrared  intensities  of  several  potential  synthetic 
precursors  to  new  polynitrogen  species  have  been 
computed  using  ab  initio  electronic  structure  theory, 
at  the  second  order  perturbation  theory  level  (MP2, 
also  known  as  MBPT(2)),  using  the  6-31  G(d)  valence 
double-zeta  polarized  basis  set.  Shown  in  Figure  1  is 
the  predicted  structure  of  triphenylmethyldiazonium 
cation,  also  known  as  trityldiazonium,  which  is  a 
possible  precursor  to  new  polynitrogen  compounds 
such  as  pentazole,  a  5-membered  ring  system.  The 
calculated  structure  shows  that  this  cation  is  unstable 


Figure  1 .  MP2/6-31  G(d)  optimized  structure  of 
triphenylmethyldiazonium  cation,  [C19N2H15]  +  .  The  predicted  C-N 
internuclear  distance  of  3.21  angstroms  indicates  that  this  cation  is 
unstable  with  respect  to  dissociation  to  triphenylmethyl  cation  and 
N2. 


“The  performance 
limits  of  currently  used 
propellants  have  been 
reached,  so  new 
energetic  compounds 
are  required  to 
significantly  improve 
the  ability  of  the 
warfighter  to  access 
and  control  space.” 
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with  respect  to  dissociation  of  N2.  Therefore,  these 
calculations  predict  that  this  cation  is  not  a  viable 
polynitrogen  precursor. 

Although  this  is  a  negative  result  in  the  sense  that  it 
indicates  that  trityldiazonium  is  not  a  stable 
precursor,  it  is  nonetheless  a  highly  useful  result  in 
multiple  ways.  First,  it  saves  significant  time  and 
effort  by  eliminating  from  consideration  subsequent 
attempts  at  synthesis  of  a  compound  that  is  not  likely 
to  be  stable.  Furthermore,  it  suggests  ways  in  which 
the  trityldiazonium  cation  may  be  chemically 
modified  in  order  to  overcome  its  instability  (e.g.,  by 
judicious  placement  of  electron-withdrawing  groups 
on  the  trityl  moeity.) 


Derivatives  of  Polynitrogen 
Precursors 

Future  work  in  this  area  will  examine  such  derivatives 
of  the  trityldiazonium  cation,  as  well  as  other  classes 
of  promising  polynitrogen  precursors. 
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Scramjet  Shock-on-Cowl-Lip  Load  Mitigation  with 
Magnetogasdynamics 

Datta  Gaitonde 

Air  Force  Research  Laboratory  (AFRL),  Wright-Patterson  AFB,  OH 


Advances  in  sustained  hypersonic  flight 

(exceeding  five  times  the  speed  of  sound) 
will  impact  future  strike  capability,  ease 
access-to-space,  and  influence  military 
strategy.  Developing  vehicles  to  survive  the  harsh 
high-speed  aerodynamic  environment  for  lengthy 
periods  poses  major  challenges.  Key  limitations  are 
imposed  by  poor  efficiency  of  current  air-breathing 
propulsion  technologies  and  by  excessive  heat 
transfer  and  drag  loads,  which  can  affect  design 
viability  and  cause  mission  failure.  Supersonic 
combustion  ramjets  or 
scramjets  will  play  a  critical 
role  by  overcoming  many  of 
these  constraints.  Effective 
design  will  require  informed 
compromise,  coupled  to 
revolutionary  flow  control 
techniques.  The  problem 
considered  here  stems  from  a 
mass-capture  standpoint; 
scramjets  function  best  when  external  compression 
shock  waves  are  focused  on  cowl  lips  (Figure  1). 
Figure  2  shows  Mach  contours  of  the  flow  pattern 
resulting  near  the  cowl  lip  without  control.  The 
crucial  feature  of  this  so-called  Edney  Type  IV 
interaction  is  the  nearly  wall-normal  supersonic  jet, 
which  impacts  the  surface.  The  stagnation  process 
yields  extremely  high,  unsteady  heat  loads,  which 
can  exceed  forty  times  nominal  values  and  lead  to 
catastrophic  failure. 

Exploring  Revolutionary  Flow 
Control  Techniques 

Traditional  approaches  to  flow  control  and  heat 
transfer  mitigation  have  not  proven  successful  at  high 
speeds.  This  has  motivated  exploration  of  more 
revolutionary  techniques.  Since  hypersonic  flow  is, 
or  can  be  artificially  made  to  be  electrically 
conducting,  an  innovative  method  calls  for  a  properly 


“These  problems 
require  months  to 
solve  on  stand-alone 
workstations  but  can 
be  solved  in  days  on 
the  high-speed  HPC 
systems." 
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Figure  1 .  Schematic  of  shock-on-cowl-lip  problem 


established  electromagnetic  field  to  provide  surface 
shielding.  The  advantages  of  magnetogasdynamics 
(MGD)  include  the  absence  of  moving  parts,  short 
control  response  times,  and  the  possibility  of 
leveraging  the  massive  scales  and  advanced 
technological  state  of  electrical  engineering.  Ground 
testing  difficulties  dictate  that  the  development  of 
these  techniques  will  be  led  by  modeling,  with 
simulations  providing  insight  into  the  flow  physics  as 
well  as  guidance  on  experimental  test-bed 
configuration. 

The  problem  is  fundamentally  multidisciplinary  since 
it  requires  integration  of  the  Navier-Stokes  and 
Maxwell  equations,  together  with  models  of  thermo¬ 
chemical,  non-equilibrium  phenomena.  This  yields  a 
large  non-linear  equation  set  and  requires  intelligent 
simplifications  for  computer  modeling.  The  current 
effort  employs  elements  of  first  principles  and 
phenomenological  approaches  for  the  MGD 
formulation.  Thus,  the  Lorentz  forces  and  energetic 
interactions  are  treated  as  source  terms;  the  electric 
field  is  obtained  by  solving  the  Poisson  equation 
arising  from  the  current  continuity  constraint  and  the 
current  is  obtained  from  the  generalized  Ohm’s  law. 
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Alleviating  High  Heat  Transfer  Rates 

Massive  computational  resources  are  mandated  by 
the  mathematical  complexity  of  the  governing 
equations  coupled  with  the  need  for  fine  spatio- 
temporal  discretizations  to  assure  solutions  of  high 
fidelity.  Despite  the  use  of  a  unique  algorithm,  which 
blends  high-resolution  upwind,  high-order  compact- 
difference/filtering  and  implicit  time-integration 
schemes,  these  problems  require  months  to  solve  on 
stand-alone  workstations  but  can  be  solved  in  days 
on  the  high-speed  HPC  systems.  This  greatly 
facilitates  numerical  exploration  of  the  parameter 
space  to  characterize  effectiveness  and  identify 
promising  candidates  for  further  study  in  ground-test 
facilities. 


Figure  3.  Effect  of  MGD-control  on  heat  transfer  rates 


To  alleviate  the  high  heat  transfer  rates,  a 
circumferential,  radially  decaying  magnetic  field  is 
imposed  on  the  flow,  consistent  with  an  axially 
directed  current.  Various  electrical  conductivity 
distributions  are  enforced  in  an  electrodeless  setup 
to  reduce  the  strength  of  the  jet  and  to  modify  the 
fluid  dynamic  bifurcation.  Figure  3  shows  the  effect 
of  one  of  the  more  promising  control  configurations 
in  which  the  primary  triple  point  is  modified  with  an 
ionized  hot-spot,  reducing  peak  heat  transfer  rate  by 
over  20%.  Surface  pressures  also  diminish  to  a 
similar  extent.  Furthermore,  parametric  studies 
demonstrate  that  these  benefits  increase  as  the 
second  power  of  magnetic  field  strength  and  first 
power  of  electrical  conductivity.  Post  processing 
yields  invaluable  insight  into  the  mechanics  of  flow 
control.  Figure  4  shows  that  the  magnetic  field  has 
modified  the  pattern  from  Type  IV  to  that  classified  as 
Type  III,  and  the  supersonic  jet  does  not  form  at  all. 
Energy  balance  analyses  indicate  a  highly  efficient 
and  reversible  flow  field  energy  extraction  process, 
which  raises  the  possibility  of  even  more  effective 
control  through  realization  of  successive  flow 
bifurcations. 


n®lc 


tdrey 
Typ®  III 
intaf»ct>oo 


Figure  4.  Effect  of  MGD-control  on  flow  field 


The  analysis  of  advanced  flow  control  techniques  is 
greatly  facilitated  by  visualizing  the  results.  Although 
the  primary  goal  is  often  to  influence  quantities  on  the 
surface  itself,  many  promising  possibilities  leverage 
the  large  gains  obtained  by  modifying  the  overall  flow 
structure.  Scientific  visualization  aids  in 
understanding  the  fluid  dynamics  and  control 
response  and  fosters  out-of-the-box  thinking. 
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Future  of  Scramjets 

This  work  clearly  shows  the  ability  of  advanced 
electromagnetic  force-based  methods  to  alleviate  or 
eliminate  problems  encountered  in  sustained 
hypersonic  flight. 

Significant  challenges  remain  both  in  enhancing  the 
theoretical  model  and  in  comprehensively  evaluating 
and  exploiting  the  potential  of  these  revolutionary 
proposals.  Phenomenological  aspects  of  the 
formulation  are  presently  being  replaced  with  first 
principles  approaches,  particularly  in  the  modeling  of 
high-temperature  effects.  The  use  of  wall  electrodes 
and  non-equilibrium  ionization  techniques  opens 
exciting  new  possibilities  in  managing  global  energy 
budgets  of  scramjet  operation.  MGD-assisted 
diffusers,  nozzles  and  combustors  promise  efficiency 
enhancements  unthinkable  with  conventional 


methods,  and  may  provide  the  key  breakthrough  to 
make  sustained  hypersonic  flight  feasible. 
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“Presto”:  Parallel  Scientific  Visualization  of  Remote  Data  Sets  by  Andrew  Johnson 

View  of  the  Presto  interface  showing  the  results  of  a  CSM  simulation.  Damage  and  material  stress  is  shown  for  a  projectile-armor 
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A  Native  American  Indian  chief  once  remarked  that  the  words  written  on  the  medallions  worn  on 
necklaces  of  his  tribesmen  are  but  a  visual  reminder  of  the  words  written  on  their  minds  and  souls.  The 
words  on  the  Air  Force  Research  Laboratory  coin  from  General  Hap  Arnold  read,  “The  First  Essential  of 
Airpower  is  Pre-Eminence  in  Research.”  The  mind  and  soul  of  America’s  transformational  department 
of  defense  exist  in  a  body  fundamentally  dependent  on  pre-eminence  in  research;  a  body  fed  by  HPC 
technology.  The  following  seven  articles  exemplify  the  importance  of  HPC  in  the  transformational 
defense  power  the  US  DoD  is  becoming. 

To  support  soldiers  in  the  field  and  scientists  in  the  laboratory,  an  effort  from  the  Army  High 
Performance  Computing  Research  Center  provides  a  persistent  data  repository  for  data  producers  and 
consumers.  The  Virtual  Data  Grid  described  in  the  article  by  Weissman  provides  just-in-time  data 
production  and  delivery  for  data  intensive  HPC  applications.  This  data  management  system 
assimilates  and  fuses  data  for  real  time  and  as-fast-as-possible  computational  systems,  pumping 
valuable  information  into  the  21st  century  battlespace.  The  Virtual  Data  Grid  may  enable  modeling  the 
evolution  of  toxic  agents  in  public  places,  providing  support  to  homeland  defense  organizations. 
Leveraging  Virtual  Data  Grid  information  technology  enables  US  and  allied  forces  to  project  power  into 
remote  environments  with  integrated  multi-source  data  access. 

The  technical  capability  discussed  by  Flemming,  Perlman,  Sarris,  Thiel,  Czarnul,  and  Tomko  in  Mixed 
EM-Circuit  Modeling  and  Parallelization  characterizes  cosite  interference  problems  relevant  to  military 
communication  ad-hoc  network  architectures.  This  modeling  tool  mixes  electromagnetic  wave 
propagation  and  circuit  modeling  equations  to  simulate  the  effects  of  collocating  electronic  equipment 
on  the  same  vehicle,  station,  or  base.  Specific  communications  impact  to  US  forces  projected 
anywhere  in  the  world  can  be  realized  by  leveraging  this  advanced  communication  focused 
information  technology. 

While  the  Virtual  Data  Grid  technology  provides  US  and  allied  forces  access  to  data  on  a  wide  area 
network  structure  and  the  cosite  electromagnetic  interference  technology  models  the  communication 
channels  connecting  the  troops  to  that  data  and  its  derivatives,  the  third  effort,  Parallel  Scientific 
Visualization  of  Remote  Data  by  A.  Johnson,  dramatically  reduces  the  necessary  network  traffic  and 
client  machine  requirements  to  get  the  data  to  the  user  by  providing  remote  parallelized  visualization  of 
large  data  sets.  This  client-server  approach  to  visualization  of  massive  data  puts  visual  data  not  only  in 
the  scientists’  hands,  but  projects  information  into  the  deployed  soldiers’  battlefield.  These  first  three 
articles  describe  rapid  advances  in  information  technology  that  connect  the  battlespace  to  large 
amounts  of  real-time  and  near-real-time  data.  Since  this  third  technology  reduces  network  traffic,  an 
alternative  application  might  enhance  space  operations  by  enabling  data  visualization  across  limited 
communication  channels. 

The  article  by  Samios  and  Welday,  Connecting  the  Navy  Research  Lab  to  the  Department  of  the  Navy’s 
Chief  Information  Officer  describes  a  successful  effort  that  physically  connects  the  two  locations  with 
asynchronous,  real  time  variable  bit  rate  transfer  capability.  This  advanced  information  technology 
employment  provides  permanent  virtual  circuits  upgrade  throughout  the  core  fabric  of  the  defense 
research  and  engineering  network  wide  area  network  (DREN  WAN).  In  addition  to  VTC  link,  this 
connection  enables  real  time  experiments  with  secondary  benefactors  including  modeling  and 
simulation,  test  and  evaluation,  and  research  and  development  users  on  the  DREN  WAN.  This  article 
details  a  terrific  example  of  leveraging  rapid  advances  in  commercially  available  network  technology  to 
improve  US  communication  connectivity. 

Included  in  this  section  is  Novel  Electron  Emitters  and  Nanoscale  Devices  by  Bernholc,  that  describes 
an  exciting  application  of  the  nation’s  HPC  resources.  The  authors  were  able  to  accomplish 
computationally  intensive  nanoscale  device  analysis  to  investigate  and  predict  the  properties  of  new 
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materials  and  technologies.  Impacts  of  this  particular  analysis  includes  flat  panel  displays  and  electron 
emitters.  This  study  from  the  Office  of  Naval  Research  provides  an  excellent  example  for  how  HPC 
plays  a  vital  role  investigating  revolutionary  concepts  in  a  field  cited  in  the  quadrennial  defense  review 
as  having  a  potentially  large  impact  on  US  military  capability. 

The  final  two  articles  describe  evolutionary  enhancements  to  two  battlefield  simulation  systems.  An 
Integrated  Missile  Defense  Approach  by  Johnson  and  Robertson  improves  the  Missile  Defense 
Agency’s  (MDA)  wargaming  capability  by  using  the  DoD  HPC  resources  in  conjunction  with  Missile 
Defense  Wargame  and  Analysis  Resource.  Under  this  effort,  the  Joint  National  Integration  Center 
(JNIC)  improved  MDA’s  homeland  defense  simulation  realism  by  increasing  the  simulation  size, 
including  fog  of  war  error  simulation,  adding  a  target  assignment  decision  aid,  and  enabling  data 
collection  in  parallel  without  performance  degradation.  The  final  article  entitled  Successfully  Operating 
WG2000  Simulation  by  Fuller  describes  a  JNIC  developed  simulation  tool,  Wargame  2000  (WG2000), 
that  leverages  HPC  capability  and  high-speed  high-volume  communication  network  technology. 
WG2000  provides  simulated  battles  for  military  planners  around  the  world  to  exercise  command  and 
control  human-in-control  analysis.  When  coupled  with  the  Virtual  Data  Grid  technology,  these 
evolutions  in  parallelization  of  wargame  simulation  tools  provide  the  next  generation  of  homeland 
defense  simulation  modeling. 

These  seven  examples  comprise  a  subset  of  HPC  projects  with  current  and  future  impact  to  the  joint 
warfighting  team.  Secretary  of  Defense  Rumsfeld  has  stated  that:  “It  is  not  only  about  changing  the 
capabilities  at  our  disposal,  but  changing  how  we  think  about  war.  Imagine  for  a  moment  that  you 
could  go  back  in  time  and  give  a  knight  in  King  Arthur’s  court  an  M-16.  If  he  takes  that  weapon,  gets 
back  on  his  horse,  and  uses  the  stock  to  knock  his  opponent’s  head,  it’s  not  transformational. 
Transformation  occurs  when  he  gets  behind  a  tree  and  starts  shooting.”  Our  nation’s  High 
Performance  Computing  resources  are  an  essential  tool  in  a  battlespace  demanding  information 
superiority  and  advanced  technology.  When  used  in  the  transformational  sense  Mr.  Rumsfeld 
describes,  this  computational  and  algorithmic  power  enables  US  forces  to  employ  an  arsenal  of 
leading  edge  technologies  in  transformational  manners. 


1st  Lt  John  Gulick,  USAF 

Air  Force  Research  Laboratory,  Sensors  Directorate,  WPAFB,  OH 

Dr,  Kueichien  Hill 

Air  Force  Research  Laboratory,  Sensors  Directorate,  WPAFB,  OH 
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The  Virtual  Data  Grid:  Just-in-time  Data  Delivery  ... 
and  Data  Production  for  HPC  Applications 

Jon  Weissman 

Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 


A  persistent  data  repository  for  data 

producers  and  consumers  is  needed  to 
support  soldiers  in  the  field  and  scientists 
in  the  laboratory.  Increasingly,  data  from  a 
large  variety  of  sources  must  be  available  quickly  and 
reliably.  Earlier  solutions  did  not  attempt  to  provide  a 
single  coherent  infrastructure  for  data  management. 
HPC  is  needed  to  produce  data  on  the  fly  such  as  in 
battlefield  simulation  scenarios.  HPC  applications 
will  also  critically  rely  on  data  products  (e.g., 
battlefield  visualization). 

Virtual  Data  Grid 

This  project  supports  and  extends  HPC  technology  to 
a  wide-area  Grid.  The  digitally  enhanced  battlefield 
of  the  future  requires  a  distributed  data  and 
computation  infrastructure.  As  shown  in  Figure  1, 
this  infrastructure  must  provide  for  the  storage  and 


delivery  of  distributed,  heterogeneous,  data  sources 
including  geographic  information  systems  data, 
weather,  sensor  grids  (e.g., 
the  Information  Grid,  NSOFT) 
and  a  variety  of  national 
assets.  Often  times,  the  data 
of  interest  is  large,  and 
distributed  across  different 
remote  storage  locations  and 
data  delivery  networks.  In 
addition,  connection  to 
remote  data  networks  may  be 
intermittent  (e.g.,  sensor  networks).  A  reliable, 
efficient,  persistent  data  network  called  the  Virtual 
Data  Grid  (VDG)  is  being  developed  for  this  purpose. 
The  VDG  allows  data  producers  and  consumers  to 
see  a  consistent  infrastructure  for  data  exchange. 

The  VDG  also  allows  data  produced  from  high 
performance  simulations  to  be  stored  and  made 


“The  digitally 
enhanced  battlefield  of 
the  future  requires  a 
distributed  data  and 
computation 
infrastructure." 


Figure  1 .  Virtual  Data  Grid 
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available  by  launching  such  simulations  on-the-fly  to 
produce  needed  data  on  demand.  A  publish- 
subscribe  applications  programming  interface  (API) 
for  data  producers  and  consumers  is  provided. 

The  VDG  leverages  existing  metacomputing  Grid 
technologies  and  solutions  to  meet  its  requirements. 
For  example,  it  uses  the  Globus  toolkit  support  to 
connect  a  grid  of  VDG  servers  that  manages  a  set  of 
persistent  storage  depots.  Internally,  the  VDG 
implements  data  scheduling  algorithms  for 
performance,  including  intelligent  caching,  staging, 
and  prefetching  of  data  based  on  application 
patterns  of  access  (an  API  for  expressing  access 
patterns  has  been  developed).  Data  scheduling 
considers  the  demand  for  a  particular  data  object  by 
concurrent  users.  Novel  data  scheduling  algorithms 
have  also  been  developed.  Data  is  moved  close  to 
the  consumer  before  the  data  is  actually  required. 
Popular  data  objects  are  moved  to  a  location  near  the 
consumer  base.  In  some  cases,  popular  data 
objects  are  replicated.  Large  data  objects  are  also 
striped  across  several  VDG  server  depots  for 
increased  performance.  Fault  tolerant  delivery  of 
data  objects  is  achieved  by  intelligent  replication 
schemes. 


simulating  the  evolution  of  toxic  agents  in  large 
public  places  (e.g.,  stadiums).  Data  from  many 
sources  must  be  integrated  and  simulations  must  be 
launched  quickly  to  solve  this  problem. 
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Environments.  One  application  is  predicting  and 


-150- 


Mixed  Electromagnetic  —  Circuit  Modeling  and 
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The  objective  of  this  research  is  to 

characterize  cosite  (collocation  of  electronic 
equipment  on  the  same  vehicle,  station,  or 
base)  interference  problems  relevant  to 
military  communication  ad-hoc  network  architectures, 
in  complex,  arbitrary  environments,  through 
scaleable  computational  techniques  for  the  numerical 
modeling  of  electromagnetic  wave  propagation 
(Maxwell’s  equations)  and  transmitter-receiver 
geometries  (circuit  equations). 

The  numerical  tools  developed  in  this  research 
provide,  for  the  first  time,  the  opportunity  of 
performing  end-to-end  characterization  of  military 
radio  network  performance  under  cosite  interference 
conditions  including  realistic  vehicle  models  and 
advanced  receiver  architectures  in  environments  as 
complex  as  a  forest.  Two  applications  evolved  from 
this  work.  The  first  application  was  a  physics-based 
optimization  of  the  system  design  (such  as  choice  of 
coding  scheme  and  power  levels).  The  second 
application  was  an  educated,  simulation-based 
planning  of  exercises  in  tactical  communication 
scenarios. 

Relying  on  Measurement-Based 
Models 

Despite  the  high  interest  that  cosite  interference 
problems  present  from  both  a  research  and  an 
application  point  of  view,  few  efforts  have  been  made 


towards  their  rigorous  theoretical  characterization. 
Straightforward  calculation 

of  interfering  power  levels  is  H - 

not  possible  because  of  the 
close  proximity  of  interferers 
and  the  presence  of 
complex  scatters  and 
antenna  platforms. 

Therefore,  some  previous 
studies  have  relied  on 
measurement-based 
models  that  omit  the 
influence  of  system-specific 
parameters  on 
communication  link 
performance.  On  the  other 
hand,  time-domain  methods 
for  the  solution  of  Maxwell’s 


“These  techniques  allow 
for  the  accurate, 
physics-based  evaluation 
of  system  performance 
via  mixed 

electromagnetic-circuit 
simulations  and  provide 
the  tactical 

communication  analyst 
with  a  reliable  simulation 
tool  that  takes  into 
account  critical 
electromagnetic 
interactions  within 
complex,  real-life 
environments." 


equations  hold  the  promise 

of  fully  accounting,  not  only  for  the  effect  of  platforms 
on  radiation  properties  of  vehicular  antennas,  but 
also  for  the  operation  of  front-end  electronics, 
through  a  state-equation  description  of  the  latter. 


The  computational  cost  of  time-domain  techniques 
grows  large,  however,  when  more  complex  operation 
environments  are  considered  for  cosite  interference 
scenarios.  An  example  of  particular  interest  is  the 
case  where  communication  between  a  receiver  and  a 
transmitter,  under  cosite  interference  conditions,  is 
attempted  within  a  forest  environment  at  very  high 
frequency  (VHF)  band  military  radio  frequencies. 
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Using  a  Method  of  Moments 
Approach 

A  hybrid  solution  has  been  implemented  for 
modeling  a  multi-antenna,  mobile  communication 
system  with  a  time-domain  technique  and  its 
interaction  with  its  forest  environment  by  means  of 
the  MoM  approach.  In  addition,  advanced  circuit 
models  are  included  through  a  recently  developed 
circuit  simulator  (TRANSIM),  that  allows  for  fast  time- 
domain  analysis  of  linear  and  nonlinear  components. 

Time  domain  electromagnetic  modeling  is  based  on 
the  Haar  wavelet  based  Multi  resolution  -  Time 
Domain  technique  (MRTD),  which  allows  for 
dynamically  adaptive  gridding  throughout  the 
computational  domain.  Using  a  novel,  efficient 
thresholding  algorithm,  wavelet  operations  are  not 
carried  out  at  regions  of  the  domain  where  slow  field 
variations  are  observed.  That  effectively  amounts  to 
establishing  a  multigrid  with  a  cell  aspect  ratio  of  1 :8 
(since  one  wavelet  level  is  used  in  all  three 
dimensions).  In  addition,  wavelet  operations  dictate 
an  adaptive,  global  repartitioning  algorithm  for 
periodic  load  balancing,  which  improves  the 
performance  of  the  parallel  code. 


Using  Communication  Scenario 
Simulations 

Communication  scenarios  have  been  simulated 
involving  the  interaction  of  neighboring  transmit- 
receive  modules,  communicating  with  a  remote  peer, 
within  a  forest  environment.  The  simulation  showed 
the  effects  of  platform,  multi-path  and  forest-induced 
attenuation  of  electromagnetic  waves  on  network 
performance  (Figure  1).  Realistic  models  of  vehicular 
platforms  have  been  incorporated  and  several 
radiation  properties  of  those  have  been  brought  to 
light  (Figure  2). 

The  US  Army  Communication-Electronics  Command 
scientists  have  predicted  interfering  power  levels  at 
multi-antenna  vehicular  systems  and  compared 
architectures  showing  the  effect  of  the  vehicle 

Figure  1 . 

Interference  pattern 
for  three  vehicular 
platforms  (shown  in 
lily)  at  50  MHz,  at  the 
plane  of  the  gap  of 
the  monopole 
antennas,  mounted 
on  them.  Thick  arrow 
indicates  the  position 
of  a  transmitting 
antenna  and  the  thin 
arrows  the  position  of 
receiving  antennas. 


Figure  2.  Electric  field 
magnitude  iso-surfaces 
on  a  military  vehicle 
staircase  model  at  50 
MHz 


positioning  on  the  performance  of  the  radio  network. 
The  results  have  highlighted  such  issues  as  the 
dependence  of  parasitic  interference  between 
neighboring  platforms  on  the  shape  of  the  vehicle 
and  geometry-specific  scattering  effects  and 
resonance. 

Scientific  visualization  was  used  to  gain  insight  into 
the  physical  phenomena  of  interest  for  this  research 
effort  and  has  led  to  several  important  conclusions 
related  to  the  effect  of  vehicle  positioning  on  radio 
network  performance  based  on  inspection  of 
visualization  tool  output. 

Implementing  Efficient,  Scaleable 
Computational  Techniques 

Efficient,  scaleable  computational  techniques  for  end- 
to-end  characterization  of  cosite  interference  in 
military  radio  networks  have  been  proposed  and 
implemented  in  this  project.  These  techniques  allow 
for  the  accurate,  physics-based  evaluation  of  system 
performance  through  mixed  electromagnetic-circuit 
simulations  and  provide  the  tactical  communication 
analyst  with  a  reliable  simulation  tool  that  takes  into 
account  critical  electromagnetic  interactions  within 
complex,  real-life  environments. 
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"Presto”:  Parallel  Scientific  Visualization  of 
Remote  Data  Sets 

Andrew  Johnson 

Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 


resto  is  an  interactive  data  visualization  tool 
developed  at  AHPCRC  which  provides 
researchers,  using  their  desktop 
workstations,  the  capabilities  of  a  large  and 
expensive  visualization  laboratory  for  visualizing  large 
numerical  simulation  data  sets.  These  data  sets  are 
computed  on,  and  reside  at,  remote  HPC  systems. 
This  is  accomplished  by  implementing  a  closely 
coupled  client-server  paradigm  that  leverages  the 
parallel  computing  technology  of  remote  HPC 
systems.  The  client  (i.e. ,  the  user’s  desktop  system) 
and  the  server  (i.e.,  the  remote  HPC  resource)  only 
need  to  be  linked  through  a  typical  Internet  or  other 
dedicated  network  connection. 

Limitations  of  Current  Computing 
Models 

A  new  visualization  tool  such  as  Presto  is  required 
due  to  various  limitations  with  current  computing 
and/or  data  processing  models.  Traditionally,  a 
researcher  would  compute  a  large  numerical 
simulation  on  a  HPC  resource  that  is  physically 
located  at  a  remote,  central  computing  center  such 
as  a  MSRC  or  DC.  They  would  download  the  entire 
completed  data  set  to  their  local  workstation  for  post¬ 
processing  and  visualization.  Today,  researchers  are 
computing  data  sets  of  between  1  and  200  gigabytes 
in  file  size  or  larger.  Transfer  of  this  amount  of  data 
across  a  network  is  impractical.  Also,  researchers  do 
not  typically  have  local  computing  resources 
powerful  enough  to  load,  process  and  visualize  the 
entire  data  set  effectively.  These  bottlenecks  have 
limited  the  size  and  scope  of  the  numerical 
simulations  that  researchers  can  create  and  process 
effectively. 


Presto  Solves  Limitations 

Presto  solves  both  of  these  problems  through  its 
client-server  implementation  and  parallel  computing 
capabilities.  The  client-server  implementation  means 
that  Presto  actually  consists  of  two  separate 
components.  The  “Presto  Client”  runs  on  the  user’s 
desktop  workstation,  either  Windows  or  UNIX-based, 
and  handles  all  user  interaction  and  displays  all 
visualization  geometry  sent  to  it 
by  the  server.  Visualization  is 
accomplished  through 
traditional  boundary  shadings, 
cross-sections,  iso-surfaces, 
streamlines  and  volume 
rendering.  The  “Presto  Server” 
runs  on  the  remote  HPC 
resource,  and  is  written  using 
the  distributed-memory  parallel 
programming  model  based  on 
MPI,  which  is  portable  to 
almost  all  HPC  systems  such  as  the  Cray  T3E,  Cray 
XI,  IBM  SP,  SGI  Origin  servers,  and  PC  clusters.  The 
server  is  responsible  for  loading  and  processing  the 
large  3-D  data  set,  and  generating  and/or  extracting 
any  visualization  geometry  requested  by  the  client 
component.  The  parallel  implementation  is  fully 
scalable,  so  the  size  of  the  data  set  that  can  be 
visualized  from  the  desktop  is  only  limited  by  the  size 
of  the  remote  HPC  resource. 

Presto  has  been  built  to  visualize  3-D  unstructured 
mesh  data  sets  composed  of  either  scalor  or  vector 
variables,  and  is  typically  being  used  to  visualize 
CFD  (Figure  1)  and  CSM  (Figure  2)  data  sets. 
Typically,  unstructured  mesh  data  sets  containing 
between  1  and  40  million  tetrahedral  elements  are 
visualized  using  up  to  32  processors  of  a  Cray  T3E. 
Data  sets  containing  over  1  billion  tetrahedral 
elements  have  been  visualized  with  Presto  using 
1024  processors.  Currently,  Presto  is  being  used  in  a 
production  environment,  and  several  future  updates 


"Presto  has  been  built 
to  visualize  3-D 
unstructured  mesh 
data  sets  composed 
of  either  scalor  or 
vector  variables,  and 
is  typically  being  used 
to  visualize  CFD  and 
CSM  data  sets.” 
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Figure  1 .  View  of  the  Presto  interface  showing  the  results  of  a  CFD 
simulation.  Velocity  vectors  are  shown  as  well  as  a  volume-rendered 
image  showing  velocity  magnitude. 


are  being  planned  which  include  implementing  a 
tighter  client-server  security  model  as  well  as 
geometry  compression  techniques  to  reduce  Presto’s 
network  bandwidth  requirements. 

Advantages  of  Presto 

Presto  provides  three  advantages  to  the  researcher: 

1)  Visualizes  remote  data  sets:  Basically,  the 
researcher  just  leaves  the  data  set  on  the  large  work 
disks  of  the  HPC  system  that  computed  the  results. 
The  “Server”  part  of  Presto  runs  on  that  HPC  system 
and  is  responsible  for  loading  and  processing  the 
numerical  simulation  results.  The  user  never  needs 
to  transfer  their  data  set  off  of  the  HPC  system  that 
generated  it.  Data  sets  located  on  any  HPC  system 
can  be  visualized  from  any  remote  location  as  long 
as  an  Internet  connection  (TCP/IP)  can  be  created 
between  the  user’s  local  workstation  and  the  remote 
HPC  system.  2)  Scalability:  The  Presto  Visualization 
“server”  is  fully  parallel  based  on  the  MPI  library.  This 
means  that  the  server  runs  on  any  parallel 


Figure  2.  View  of  the  Presto  interface  showing  the  results  of  a  CSM 
simulation.  Damage  and  material  stress  is  shown  for  a 
projectile-armor  penetration. 


architecture  and  is  fully  scalable  in  terms  of  memory. 
When  visualizing  a  small  data  set,  the  user  can  use 
only  a  few  processors  to  load  the  data  set.  When 
visualizing  a  large  data  set,  the  user  may  have  to  use 
more  processors.  Typically,  around  8  to  16 
processors  are  used,  but  Presto  has  been  tested  with 
as  many  as  1 024  processesors  on  a  Cray  T3E.  This 
many  processors  were  required  to  visualize  one  of 
the  largest  data  sets  ever,  which  involved  an 
unstructured  mesh  containing  1  billion  tetrahedral 
elements.  3)  Easy-to-use:  The  “client”  part  of  Presto 
handles  all  of  the  GUI  user  interface  and  displays  all 
of  the  graphics  components  sent  to  it  by  the  server 
over  the  Internet.  The  client  runs  on  Windows  or  any 
UNIX  flavor  (Linux,  SGI,  etc.).  Once  the  connection 
between  the  client  and  remote  server  is  made,  the 
application  behaves  as  any  traditional  desktop 
application. 

Presto  saves  time  and  money  (i.e.,  users  can  use 
their  traditional  workstations  and  not  an  expensive 
visualization  system)  and  effectively  uses  HPC 
parallel  resources  for  post-processing  and 
visualization.  A  user  from  almost  any  type  of 
desktop  system  can  visualize  data  sets  of  almost  any 
size  quickly  and  efficiently.  This  makes  the  use  of 
numerical  simulation  by  researchers  more  effective, 
efficient,  and  easier,  especially  for  large  data  sets. 
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Connecting  the  Naval  Research  Laboratory  to  the 
Department  of  the  Navy’s  Chief  Information 
Officer 

John  Samios  and  Alan  Welday 

High  Performance  Computing  Modernization  Program  (HPCMP),  Arlington,  VA 


Just  prior  to  September  1 1 , 2001 ,  the 

Department  of  Navy  (DON)  Chief  Information 
Officer  (CIO)  approached  the  Network 
Infrastructure  Services  Agency  (NISA)  to 
support  a  high-speed  permanent  virtual  path  (PVP) 
between  their  offices  and  the  Defense  Research  and 
Engineering  Network  (DREN)  node  in  the  Pentagon 
(NISA  supports  both  locations  and  the  cross- 
connection).  After  September  1 1 , 2001 ,  available 
bandwidth  on  some  of  the  links  was  reduced  and  the 
original  request  could  no  longer  be  easily 
accommodated.  As  a  result,  the  request  for  a  40- 
Mbps  PVP  was  reduced  to  6  Mbps.  The  Naval 
Research  Laboratory  (NRL)  looked  to  DREN  to 
provide  the  link. 

The  technical  problem  centered  around  how  to 
connect  multiple  DoD  sites  using  real-time  variable 
bit  rate  virtual  circuits,  which  have  very  low  latency 
and  jitter.  Resolution  of  the  problem  will  have  a  wider 
impact  to  DoD  Science  and  Technology,  Modeling 
and  Simulation,  and  Test  and  Evaluation  communities 
within  the  Navy. 

Early  Issues 

There  were  several  early  issues  that  were  identified: 
physical  connectivity  from  the  closest  NISA 
Asynchronous  Transfer  Mode  (ATM)  switch  to  the 
DREN  ATM  switch  at  the  Pentagon,  availability,  cost, 
and  characteristics  of  the  PVP  had  to  be  determined. 
Security  also  had  to  be  considered;  the  existing 
security  on  the  DON  CIO  workstations  could  not  be 
compromised.  Any  traffic  between  these 
workstations  and  the  NISA-supported  systems  would 
leave  PT-1 ,  tunnel  through  to  the  DREN  wide  area 
network  (WAN),  enter  the  NISA  enclave  and  existing 
NISA  security  mechanisms  through  the  same  path 
that  Unclassified  but  Sensitive  Internet  Protocol 


Router  Network  and  Internet 
traffic  follows,  without 
interacting  with  the  NISA 
infrastructure. 

In  the  past,  procuring  many 
independent  unclassified 
and  classified  connections 
between  each  site  would 
have  solved  this  problem. 

But  separate  circuit 
solutions  are  very  difficult  to 
manage,  very  expensive, 
and  usually  result  in  severe  limits  on  bandwidth 
availability. 

Connection  Established 

In  September  of  2002,  a  solution  was  completed  that 
connects  several  TeraMedia  VTC  sessions  between 
NRL  and  DON  CIO  spaces.  This  was  done  by  using 
real-time  variable  bit  rate  (rt-VBR)  setups  over  DREN. 
Figure  1  shows  the  connection  using  ATM-connected 
Macinstosh  workstations  and  a  Sun  server  at  DON 
CIO  (PT-1). 

This  project  brought  to  the  fore  other  ATM 
requirements  in  the  HPMCP  community  for  rt-VBR 
permanent  virtual  circuits  (PVCs).  As  a  result,  DREN 
WAN  switches  throughout  the  core  fabric  were 
upgraded  to  support  rt-VBR  PVCs  as  well  as  switched 
virtual  circuits. 

The  Future  of  ATM  and  the  Use  of 
PVCs  With  DREN 

Even  as  long  haul  WAN  providers  move  away  from 
ATM  based  networks  to  increasingly  sophisticated  IP 
based  networks,  there  will  be  ATM-oriented  Quality  of 
Service  requirements  that  will  be  difficult  to  meet  with 


"In  September  of  2002,  a 
solution  was  completed, 
that  connects  several 
TeraMedia  VTC  sessions 
between  NRL  and  DON 
CIO  spaces.  This  was 
done  by  using  real-time 
variable  bit  rate  (rt-VBR) 
setups  over  DREN.” 


-155- 


-  Leveraging  Information  Technology  - 

the  new  architectures  for  some  time  to  come.  At  the 
start  of  this  project,  DREN  could  not  handle  the  low 
jitter  requirements  that  drove  a  rt-PVCs  solution; 
today  it  can.  The  beneficiaries  of  this  innovation  are 
not  only  the  original  testers  but,  a  much  wider  group 
of  communities  that  includes  modeling  and 
simulation,  test  and  evaluation,  and  research  and 


development.  Today,  “hardware-in-the-middle”  or 
“man-in-the-loop”  simulations  and  tests  can  be  done 
in  real-time.  Without  this  capability,  the  experiments 
cannot  be  done  remotely,  and  assumptions  had  to  be 
made  by  researchers  to  simulate  the  “real-time” 
effect  put  on  their  test  or  experiment. 


jJQklCtOVTC  TQDREH 
PROJECT 


Figure  1 .  The  box  on  the  left  shows  an  ATM  switch  in  the  PT-1  wiring  closet  under  N1SN  control.  From  the  PT-1  ATM  switch,  there  is  a  6-Mbps 
rt-VBR  PVP  through  the  NISN  cloud  to  the  DREN  switch  in  the  Pentagon.  The  PT-1  switch  signals  with  the  DREN  switch  (and  is  using  a  subset  of 
its  ATM  NSAP  address  space).  The  DREN/PT-1  switches  fully  signals  with  DREN  so  direct  end-to-end  ATM  connection  are  possible.  The  LECS, 
LES,  and  BUS  are  running  on  the  DREN  switch  and  an  ELAN  was  created  for  this  project. 
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Novel  Electron  Emitters  and  Nanoscale  Devices 

Jerry  Bernholc 

North  Carolina  State  University,  Department  of  Physics,  Raleigh,  NC 
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anotubes  are  one  of  the  most  interesting 
new  materials  to  emerge  in  the  past 
decade.  Several  layered  materials  form 
tubular  structures,  but  the  most  prevalent 
are  carbon  (C)  and  boron  nitride  (BN)  nanotubes. 

The  discovery  of  carbon  nanotubes  as  a  material  with 
outstanding  mechanical  and  electrical  properties  has 
led  to  a  quest  for  other  novel  graphene-based 
structures  with  technologically  desirable  properties. 
The  closely-related  boron  nitride  (BN)  nanotubes  and 
mixed  BN/C  systems,  which  are  now  being  produced 
in  gram  quantities,  have  electronic  properties  that  are 
complementary  to  pure  carbon  nanotubes  and  could 
thus  be  useful  in  a  variety  of  electronic  devices. 

The  focus  of  this  work  was  to  use  Challenge-level 
HPC  resources  to  investigate  and  predict  properties 
of  advanced  new  materials  and  technologies  critical 
to  the  DoD’s  needs.  This  research  has  shown  that 
BN/C  nanotube  structures  can  be  used  for  effective 
band-offset  engineering  in  nanoscale  devices,  and 
for  robust  field  emitters  with  efficiencies  of  up  to  two 
orders  in  magnitude  higher  than  those  built  from 
carbon  nanotubes. 

Using  Sophisticated  ab  initio 
Techniques 

Until  very  recently,  the  design  and  prediction  of 
advanced  materials  has  been  an  empirical 
undertaking.  The  advent  of  HPC  enabled  the  use  of 
sophisticated  ab  initio  techniques,  which  can  predict 
the  properties  of  complex  materials.  The  simulations 
were  carried  out  using  a  novel,  real-space  multigrid 
method  developed  especially  for  very  large-scale 
quantum  calculations.  Plane-wave-based  methods 
have  also  been  used  for  independent  crosschecking 
of  the  key  results.  The  use  of  multigrid  techniques 
permits  efficient  calculations  on  ill-conditioned 
systems  with  multiple  length  scales  and/or  high- 
energy  cutoffs.  The  multigrid  codes  have  been  very 
well  optimized  for  massively  parallel  computers  and 


their  execution  speed  scales  linearly  with  the  number 
of  processors.  The  calculations  required  up  to  128 
processors  of  the  Cray  T3E  supercomputer.  The 
calculations  were  carried  out  in  a  supercell  geometry, 
using  up  to  6  layers  of  boron  nitride  and  carbon.  In  a 
periodic  super  lattice  of  polar  and  non-polar  materials 
(such  as  BN  and  C),  any 
spontaneous  polarization 
field  manifests  itself  as  a 
periodic  oscillation  in  the 
electrostatic  potential  of  the 
system,  due  to  periodic 
boundary  conditions.  The 
full  potential  then  displays  a 
saw  tooth  behavior  that  is 
the  signature  of  a 
spontaneous  polarization 
field  superimposed  onto  a 
periodic  crystalline  potential. 

In  order  to  obtain  the  value 
of  this  field,  the  planar  average  of  the  electrostatic 
potential  along  the  nanotube  axis  has  been 
evaluated.  This  average,  which  is  over  the  plane 
perpendicular  to  the  tube,  displays  strong  oscillations 
due  to  the  varying  strength  of  the  ionic  potentials.  To 
subtract  out  this  effect,  the  corresponding  “bulk” 
averages  have  been  subtracted  out,  and  thereby  the 
one-dimensional  macroscopic  average  of  the 
electrostatic  potential  of  the  system  was  calculated. 
Since  nanotube  calculations  include  a  sizable 
vacuum  region  surrounding  the  nanotube,  the  band 
offsets  could  be  computed  directly  by  referring  the 
highest  occupied  levels  to  the  vacuum  level. 

Symmetry  of  Nanotube  Important 

It  is  clear  that  symmetry  of  the  nanotube  plays  an 
important  role  in  determining  the  magnitude  of  the 
spontaneous  polarization  field.  The  strongest  effects 
will  be  observed  in  the  (k,  0)  zigzag  nanotubes,  since 
this  geometry  maximizes  the  dipole  moment  of  the 
BN  bond.  This  study  has  primarily  concentrated  on 


“This  research  has  shown 
that  BN/C  nanotube 
structures  can  be  used  for 
effective  band-offset 
engineering  in  nanoscale 
devices,  and  for  robust 
field  emitters  with 
efficiencies  of  up  to  two 
orders  in  magnitude 
higher  than  those  built 
from  carbon  nanotubes.’’ 
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zigzag  nanotubes  with  diameters  of  up  to  12  E.  The 
value  of  the  polarization  field  obtained  from  the  slope 
of  the  macroscopic  average  potential  for  zigzag 
tubes  is  shown  in  Figure  1 .  In  contrast,  the  (k,  k) 
armchair  nanotubes  are  not  expected  to  display  any 
spontaneous  polarization  field.  This  is  because  any 
individual  nanotube  ring  must  be  charge  neutral,  so 
that  no  fields  are  possible.  This  has  been  directly 
verified  for  the  case  of  a  (5,5)  BN/C  super  lattice. 


Figure  1 .  The  structure  of  a  BN/C  nanotube  superlattice  (top),  the 
average  electrostatic  potential  (blue)  and  its  macroscopic  average 
(red).  The  slope  in  the  macroscopic  potential  leads  to  an  electric 
field  inside  the  nanotube. 

Chiral  nanotubes  will  have  fields  that  are  between  the 
zigzag  and  armchair  values. 

From  the  calculated  band-offsets,  it  is  clear  that  the 
valence  state  of  the  BN/C  super  lattices  is  always 
spatially  localized  in  the  carbon  region.  This  state 
plays  an  important  role  in  determining  the  response 
to  the  macroscopic  electric  field  induced  by  the  BN 
section.  In  particular,  the  oscillatory  behavior 
observed  as  a  function  of  the  helicity  index  can  be 
understood  from  the  symmetry  property  of  the 
valence  state,  displayed  in  Figure  2  for  two  zigzag 
nanotubes.  Figure  2  directly  shows  that  the  valence 
state  is  localized  within  the  carbon  section.  For  small 


diameter  nanotubes,  it  displays  either  a  longitudinal 
or  a  transverse  symmetry  as  exemplified  by  the  (7,0) 
and  (8,0)  tubes,  respectively.  Specifically,  for  (k  = 
3p+1 , 0)  nanotubes  such  as  the  (7,0)  tube,  the 
valence  state  always  assumes  a  longitudinal 
character.  The  axial  distribution  of  valence  electrons 
will  therefore  induce  a  depolarization  field  that  is 
opposite  to  the  one  intrinsic  to  the  BN  section.  This 
field  is  quite  effective  in  small  diameter  nanotubes  in 
reducing  the  polarization,  so  that  for  the  (7,0) 
nanotube  the  screening  is  almost  total  and  the 
observed  macroscopic  field  is  close  to  zero.  For 
larger  diameter  nanotubes,  the  symmetry  of  the 
valence  state  gradually  loses  its  peculiar  axial  or 
longitudinal  character  and  the  macroscopic  field 
asymptotically  approaches  the  value  obtained  for  a 
flat  BN  sheet  in  the  “zigzag”  direction. 

Assessing  the  Effectiveness  of  BN/C 
Systems 

Despite  the  screening  by  the  valence  electrons 
distributed  over  the  carbon  portion  of  the  BN/C 
superlattices,  a  net  polarization  field  is  built  up  along 
any  zigzag  structure.  The  existence  of  an  intrinsic 
macroscopic  field  will  clearly  influence  the  extraction 
of  electrons  from  BN/C  systems,  when  these  are 
used  as  field  emitting  devices.  Qualitatively,  a  good 
electron  emitter  is  characterized  by  a  large 
geometrical  field  enhancement  factor,  due  to  the 
needle-like  shape  of  nanotubes,  and  a  small  work 
function  that  is  an  intrinsic  property  of  the  emitter 
material  (Figure  3).  Quantitatively,  the  current  at  the 
emitter  tip  is  calculated  from  the  so-called  Fowler- 
Nordheim  relationship.  To  assess  the  effectiveness  of 
BN/C  systems  as  emitters,  the  field  enhancement 
factor  for  tubes  of  length  15.62  E  and  an  applied  field 
of  0.1 1  V/E  have  been  computed.  As  expected,  the 
applied  field  is  very  well  screened  inside  the  system 
and  the  local  field  enhancement  factors  were  similar 
for  different  BN/C  systems.  Since  the  field 
enhancement  factor  increases  linearly  with  the  size  of 


Figure  2.  Electron  distribution  in  the  top  valence  state  for  (7,0)  and  (8,0)  BN/C  nanotubes.  Note  that  the  symmetries  of  the  two  states  are  quite 
different. 
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Figure  3.  Electron  distribution  near  a  tip  of  a  BN/C  nanotube  in  a  field 
emitter  configuration 


up  to  two  orders  in  magnitude  as  compared  to  pure 
carbon  nanotube  systems. 

Future  Uses  for  Nanotubes 

Nanotubes  are  prime  candidates  for  novel  electron 
emitters,  to  be  used  in  ultrahigh  resolution  flat  panel 
displays  and  cold-cathode-based  microwave 
amplifiers.  The  hundred-fold  increase  in  the 
emission  current  density  would  obviously  have  a 
major  effect  on  the  use  and  efficiency  of  electron 
emitters  at  the  battlefield  and  in  support  systems. 

The  variability  of  the  band  offsets  in  BN/C  systems  as 
a  function  of  their  radius  offers  substantial  flexibility 
when  designing  novel  nanoscale  electronic  devices. 


the  nanotubes,  large  enhancement  factors  are 
possible  for  long  nanotubes.  However,  the 
polarization  field  can  clearly  be  used  to  further 
enhance  the  field  emitting  properties.  In  order  to 
examine  this  effect  quantitatively,  researchers  have 
built-up  a  set  of  finite  sized  (6,0)  zigzag  structures 
using  B,  N,  and  C  in  various  combinations.  The  work 
functions  were  then  computed  as  the  difference 
between  the  vacuum  level  and  the  Fermi  energy  of 
the  system.  The  former  was  obtained  by  the 
previously  discussed  potential  average  procedure, 
while  the  latter  was  simply  given  by  the  highest 
occupied  eigen  state-of-the-system.  With  this 
method,  we  calculated  a  work  function  of  5.01  eV  for 
a  (10,10)  carbon  nanotube,  in  close  agreement  with 
previously  published  values.  For  the  smaller 
diameter  (6,0)  carbon  nanotube,  we  found  a 
considerably  larger  work  function  of  6.44  eV.  This  is 
primarily  due  to  curvature  effects. 

Turning  to  the  heterostructures,  the  researchers 
considered  a  finite  NB/C  (6,0)  system  consisting  of 
four  alternating  N  and  B  layers  followed  by  8  C 
layers,  giving  a  total  of  96  atoms.  Due  to  the  net 
polarization  field  experienced  by  the  electron,  the 
work  function  is  reduced  to  5.04  eV  at  the  C  tip  and 
increased  to  7.52  eV  on  the  N  tip.  The  same  trend  is 
observed  for  the  BN/C  system,  for  which  the  work 
function  on  the  B  tip  is  equal  to  5.00  eV  while  it  takes 
a  value  of  6.45  eV  on  the  C  tip.  Therefore,  the  work 
function  decreases  by  a  significant  1 .40  eV,  as 
compared  to  the  pure  carbon  system.  This  is  large 
enough  to  lead  to  significant  macroscopic  effects. 
According  to  the  Fowler-Nordheim  relationship,  the 
current  density  depends  exponentially  on  the  work 
function.  It  follows  that  the  insertion  of  BN  segments 
in  C  nanotubes  will  increase  the  current  density  by 
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An  Integrated  Missile  Defense  Approach 
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The  Missile  Defense  Agency  (MDA)  uses  a 
series  of  operator-in-the-loop  wargames  to 
examine  Ballistic  Missile  Defense  System 
(BMDS)  issues  for  the  near-term  (2004-2006) 
and  far-term  (2012).  The  primary  means  of 
examining  these  issues  is  to  use  DoD  HPC  resources 
in  conjunction  with  Missile  Defense  Wargame  and 
Analysis  Resource  (MDWAR)  and  other  federated 
simulations.  To  achieve  optimal  realism  within  its  war 
games,  MDWAR  explicitly  models  threats, 
interceptors,  sensors,  and  communication  systems 
(including  passing  tactical  message  contents  in  near¬ 
native  formats).  This  approach  permits  MDWAR  to 
enhance  field  exercises  and  other  HWILT  by 
providing  high-level  integration  between  simulated 
future  missile  defense  system  elements  and  today’s 
real  battle  management,  command,  control, 
communications,  computers  and  intelligence 
equipment.  Planners  can  evaluate  the  practical 
effectiveness  of  adding  an  additional  family  of 
systems  interoperability  features  by 
extending  the  tactical  messages  set 
within  MDWAR’s  joint  data  network 
communications  model. 


MDWAR  Innovations 

MDWAR’s  design  includes  extensive 
data  collection  and  information 
retrieval  capabilities  that  provide 
insight  into  the  tactical  messages  and 
engagement  event  outcomes.  The 
data  collected  during  a  wargame  are 
available  worldwide  using  the  Secret 
Internet  Protocol  Routed  Network 
(SIPRNET)  and  any  commercially 
available  Internet  browser.  Multiple 
innovations  are  a  direct  result  of  the 
HPC  contributions.  One  innovation  is 
that  MDWAR  is  the  first  successful 
Optimistic,  Parallel,  Discrete  Event 
Simulation  (O-PDES)  framework.  In 
achieving  MDWAR’s  vision,  the 
designers  committed  to  the  use  of  an 


O-PDES  framework  as  the  basic  simulation  engine. 
The  decision  to  use  O-PDES  as  the  foundation  for  a 
real-time,  interactive,  simulator  destined  to  participate 
or  even  lead  Distributed  Interactive  Simulations  and 
High  Level  Architecture  federations  is  not  without 
controversy.  Synchronous  Parallel  Environment  for 
Emulation  and  Discrete-Event  Simulation  (SPEEDES) 
is  the  most  significant  advancement  in  state-of-the-art 
discrete  event  simulation  in  the  last  five  years  and 
was  the  only  mature  framework.  Its  primary  use  is  in 
as-fast-as-possible  (AFAP)  academic  applications  but 
it  is  available  as  a  government  off-the-shelf  product. 

MDWAR  expands  the  scalability  envelope.  The 
primary  reason  for  replacing  the  previous  wargaming 
tool  is  to  achieve  the  performance  needed  to 
simulate  more  objects  at  higher  fidelity.  In  the  past, 
problem  size  and  fidelity  were  reduced  to  achieve 
real-time  execution.  MDWAR’s  goal  is  to  maintain  the 
desired  problem  size  and  fidelity  and  “dial-up”  the 
computer  power  to  achieve  real-time  performance 


J3MDS  Global  Network 


Figure  1 .  BMDS  System 
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through  the  use  of  distributed  processing 
technologies.  The  first  National  Missile  Defense 
wargame  in  1999  was  run  on  twelve  CPUs  and 
involved  six  radars,  a  simple  point-to-point 
communications  model,  tens  of  threat  objects,  tens  of 
interceptors  and  25  non-interactive  player  positions. 
Seven  months  after  its  implementation,  MDWAR  was 
simulating  56  radars,  3,000  interceptors,  50  threat 
objects,  59  fully  independent  battle  managers,  and  a 
theater-wide  Joint  Tactical  Information  Distribution 
System  (JTIDS)  communication  system  operating  in 
an  imperfect  space  track  environment.  It  supports  32 
fully  interactive  player  positions  at  three  command 
and  control  levels  and  runs  on  25  CPUs.  Two  months 
later,  MDWAR  was  handling  an  approximately  50 
percent  increase  in  the  threat  load  through  the 
modeling  of  duplicate  tracks  in  an  imperfect  Single 
Integrated  Air  Picture.  Test  problems  are  being  run 
on  88  CPUs  in  preparation  for  even  larger  problems. 
The  practical  demonstration  of  this  kind  of  scalability 
is  another  important  advancement  in  the  state-of-the- 
art. 

O-PDES  Techniques  Eliminates 
Simulation  Constraints 

MDWAR  changes  the  fundamental  nature  of  the 
capability  versus  performance  tradeoff  in  force-on- 
force  level,  real-time  simulators.  With  previous 
missile  defense  simulators,  it  was  often  necessary  to 
simplify  the  simulation  to  achieve  real-time  execution 
by  reducing  modeling  detail  and  resolution  or 
scenario  complexity.  MDWAR  pioneers  the 
application  of  O-PDES  techniques  to  harness  the 
processing  capability  of  today’s  high-performance, 
multi-processor  computers.  MDWAR’s  advanced 
technology  all  but  eliminates  the  need  to  constrain 
the  simulated  scenario  definition  for  real-time 
execution.  MDWAR  maintains  real  time  performance, 
simulates  more  defense  elements  at  higher  fidelity, 
and  operates  in  more  complex  and  realistic 
environments  than  previously  possible  with  other 
wargaming  tools.  Its  high-end  capabilities  are 
impressive;  yet  it  retains  its  predecessor’s  variable 
fidelity  and  portability. 

MDWAR  overcomes  the  customary  constraint  to 
model  the  perfect  information  situation.  In  a  Joint 
Theater  and  Missile  Defense  Organization  wargame, 
the  JNIC  did  test  the  introduction  of  fog-of-war 
contributors  to  evaluate  joint  command  and  control 
modes  and  firing  doctrine  in  the  presence  of 
imperfect  information.  Real-world  location  and 


alignment  biases  (errors)  were  intentionally 
introduced  into  MDWAR  sensor  models;  the 
perceived  tracks  were  passed  to  a  simulated  JTIDS 
communications  network  where  explicit  correlation 
algorithms  produced  realistic  track  “drops,”  “swaps,” 
and  “duals”  as  well  as  “Reporting  Responsibility 
dueling”  behaviors.1  The  sensor  biases  and 
correlation  control  parameters  were  tuned  to  achieve 
the  target  space  track  picture  quality  necessary  to 
satisfy  the  game  objectives.  This  represents  an 
advance  in  the  state-of-the-art  because  it  is  the  only 
real-time,  interactive  simulator  with  both  the  fidelity 
and  capacity  to  look  at  the  theater-wide  ballistic 
missile  defense  operational 
behaviors  in  the  presence  of 
uncertain  information. 

Target  Assignment 

Decision  Aid 

Developed 

MDWAR  also  provides  a  test 
environment  within  which  a 
variety  of  test  articles, 
concepts  of  operations 
(CONOPS),  tactics, 
techniques  and  procedures, 
algorithms,  decision  aids,  real 
world  systems,  and 
prototypes  are  embedded 
and  their  human  in  control 
aspects  analyzed.  In  an  early 
application,  MDWAR 
developers  were  asked  to 
automate  the  specified  joint 
firing  doctrine  in  a  way  that 
allows  the  operators  to  control  the  theater  ballistic 
missile  engagements  using  a  management  by 
exception  approach.  By  designing  the  Automated 
Weapon  Target  Assignment  to  be  fully  compatible 
with  MDWAR’s  mission  (e.g.,  operate  in  a  fog-of-war 
environment),  the  developers  have  produced  a 
prototype  of  what  is  believed  to  be  the  first  theater¬ 
wide  automated  weapon  target  assignment  decision 
aid.  This  first  generation  prototype  was  successfully 
exercised  in  an  imperfect  information  environment  in 
its  very  first  exposure. 

MDWAR  has  a  heavy  data  collection  concept  that 
eliminates  performance  impacts.  The  primary 
purpose  of  any  simulator  is  to  generate  data  and 
MDWAR  is  no  exception.  It  is  common  practice  on 


"With  previous  missile 
defense  simulators,  it 
was  often  necessary  to 
simplify  the  simulation 
to  achieve  real-time 
execution  by  reducing 
modeling  detail  and 
resolution  or  scenario 
complexity.  MDWAR 
pioneers  the  application 
of  O-PDES  techniques 
to  harness  the 
processing  capability  of 
today’s  high- 
performance,  multi¬ 
processor  computers. 
MDWAR’s  advanced 
technology  all  but 
eliminates  the  need  to 
constrain  the  simulated 
scenario  definition  for 
real-time  execution.” 


1  R2  Dueling  refers  to  reporting  rules  for  radar  data.  Conflicts  arise  in  radar  data  coming  in  and  they  must  be  deconflicted. 
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many  (even  non-real-time)  simulators  to  turn  off  data 
collection  in  an  effort  to  get  acceptable  runtime 
performance.  MDWAR  has  the  goal  of  collecting 
data  with  little  or  no  impact  on  runtime  performance. 
MDWAR’s  architecture,  which  consists  of  a 
combination  of  shared  memory  buffering  and  a 
separate  dedicated  data  collection  application, 
satisfies  this  goal.  The  contents  of  every  tactical 
message  and  data  on  every  simulated  event 
(typically  one  to  two  million)  produce  one  to  two 
gigabytes  of  data  that  are  collected  in  real-time  with 
no  measurable  negative  impact  on  runtime 
performance. 

Technical  Enhancements  Spark 
Improvements 

Through  the  use  of  HPC  and  state-of-the-art 
advances  in  simulation,  MDA  and  the  HPCMP  have 
taken  wargaming  to  a  new  level.  MDWAR’s  design 
goal  is  to  examine  and  to  develop  CONOPS, 
doctrine,  tactics,  techniques  and  procedures  through 
the  use  of  operator-in-the-loop  experiments  that 
include  real-world  fog-of-war  challenges.  The 
combination  of  advanced  simulation  technologies, 
performance  oriented  architecture,  state-of-the-art 
HPC  computers,  and  unique  gaming  facilities  put  this 
goal  well  within  the  reach  of  the  air  and  missile 
defense  communities.  These  technical 
enhancements  translate  into  a  number  of 
improvements  in  the  support  provided  to  the  MDA 


wargaming  customers:  a  dramatic  increase  in  the 
numbers  of  runs  during  game  times;  significant 
improvement  in  quantitative  data  collection  leading  to 
better  analysis  and  understanding  of  emergent 
behaviors;  and  a  rapid  prototyping  laboratory  for 
analyzing  operational  concepts,  battle  management, 
mission  function  automation,  and  operator  displays. 
The  end  result  is  a  flexible  tool  that  can  be  readily 
configured  to  represent  desired  modes  of  command 
and  control,  system  architecture,  and  threat. 
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The  JNIC  provides  the  nation  a  center  of 
excellence  for  joint  missile  defense 
interoperability  testing,  wargaming,  exercises 
simulation,  modeling,  and  analysis.  JNIC 
uses  a  locally  developed  simulation,  Wargame  2000 
(WG2000).  WG2000  is  intended  to  provide  a 
simulated  combat  environment  that  will  allow 
warfighting  commanders,  their  staffs,  and  the 
acquisition  community  to  examine  missile  and  air 
defense  CONOPS.1  In  the  AFAP  mode  (non-real- 
time),  WG2000  has  a  significant  economic  advantage 
in  executing  test  simulations  in  parallel  on  the  Origin 
2000  servers  instead  of  the  serial  execution  seen  on 
most  real-time  programs.  The  Air  and  Missile 
Defense  Simulation,  or  AMDSim,  is  the  core  software 
computational  component  of  WG2000.  AMDSim 
contains  the  code  for  all  functional  simulation 
elements  including  battle  management,  sensors, 
communications  nodes,  missile  threat  and  defense 
elements,  and  graphics  displays.  Using  WG2000, 
military  planners  from  around  the  world  exercise 
command  authority  in  simulated  battles  for  command 
and  control  (C2)  human-in-control  analysis. 

SPEEDES  Provides  Wargame  2000 
Framework 

Jet  Propulsion  Laboratory’s  state-of-the-art  SPEEDES 
framework  forms  the  underlying  basis  for  WG2000’s 
high-performance,  object-oriented,  discrete-event, 
real-time  simulation  support.  With  its  forte  in  time  and 
multi-CPU  management,  SPEEDES  supports  both 
conservative  and  optimistic  synchronization 
strategies.  WG2000  improves  upon  the  previous 
generation  MDA/JNIC  Advanced  Real-time  Gaming 


Universal  Simulation.  This  is  a  pursuit  for  higher 
fidelity,  greater  realism,  and  configuration  flexibility 
with  significant  parallel  CPU  execution  supported  by 
meaningful  graphics.  JNIC  human-in-control  C2 
wargame  pioneering  efforts 
remain  a  challenging  effort. 

A  WG2000  simulation 
execution  results  in  a  35  to  60 
minute  consumption  of  from  3 
to  25  CPUs  in  either  dedicated 
or  shared  modes.  Most 
executions  use  22-25  CPUs  in 
shared  mode.  In  the  shared 
mode,  up  to  six  complete 
copies  of  the  simulation  may 
execute  simultaneously  per  server.  WG2000 
simulation  executions  most  frequently  occur  in  the 
AFAP  mode  on  the  shared  servers.  WG2000  does 
not  require  the  use  of  special  hardware  such  as 
aircraft  cockpits  or  communication  devices  often 
seen  in  other  simulator  types. 

WG2000  server  utilization  nearly  doubled  in  2001 2. 
Such  a  huge  execution  success  on  HPC  servers 
cannot  occur  with  a  traditional  real-time  simulation 
product  due  to  serial  operation  dependencies. 
Consequently,  with  this  economic  model,  the  MDA 
program  realizes  extraordinary  execution  productivity 
that  enhances  the  quality  of  the  delivered  software. 

Parallelization  Offers  Time  Savings 

Large-scale  servers  seen  today  with  anywhere  from 
256  to  51 2  CPUs  at  the  MSRCs  and  up  to  1 28  CPUs 
at  the  JNIC  DC  offer  their  respective  sites  the 
capacity  to  scale  their  products  with  the  use  of  more 


“In  2001 ,  WG2000 
achieved  success  at 
the  eleven  CPU  scaling 
level.  Today,  scaling 
from  22  to  25  CPUs 
represents  more  than 
a  100  percent  scaling 
improvement." 


1  http://WWW.JNTF.  OSD.  MILVpROGRAMS/mODELING/WG2K.  ASP 

2  Approximately  10,000  simulation  executions  occurred  in  2000  and  19,144  occurred  in  2001,  a  performance  level  enhanced  by  improved 
communication  and  a  server  upgrade.  Up  to  30,000  may  be  achieved  in  2003. 
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CPUs  applied  in  parallel  to  the  problem  space.  In 
effect,  a  problem  that  started  as  a  serial  sequence  of 
events,  becomes  divided  into  parallel  tasks  that 
execute  on  either  dynamically  or  statically  assigned 
CPUs.  The  parallel  architecture  model  should 
produce  more  work  in  the  same  period  of  time,  when 
compared  with  the  serial  model.  However,  the 
launch,  detection,  and  defense  against  a  threat 
missile  obviously  present  a  serial  time  sequence  of 
events.  Consequently,  the  composition  of  parallel 
tasks  demands  much  from  the  software  designers.  In 
2001 ,  WG2000  achieved  success  at  the  eleven  CPU 
scaling  level.  Today,  scaling  from  22  to  25  CPUs 
represents  more  than  a  100  percent  scaling 
improvement. 


The  combination  of  the  WG2000  execution  model 
and  large-scale  servers  provide  the  MDA  with  a 
successful,  low-cost  per  execution  model  through  the 
unique  application  of  large  server  technology. 
WG2000  expects  to  expand  to  even  faster  large-scale 
servers  to  produce  even  more  product  during  its  life 
cycle. 

References 

SPEEDES  User’s  Guide,  #  S024,  Rev.  3,  4,  Solana  Beach, 
CA.  Metron  Inc.,  October  2001. 

CTA:  Forces  Modeling  and  Simulation/C4l 
Computer  Resources:  SGI  Origin  2000  [JNIC] 
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High  Performance 
Computing 
Modernization 
Program 


Introduction 


The  DoD  HPCMP  is  a  strategic  national  defense  initiative  that  exists  to  enable  over  4,000  scientists 
and  engineers  to  address  the  engineering  challenges  of  the  S&T  and  T&E  communities.  The 
program  provides  an  unparalleled  collaborative  environment,  linking  users  at  over  100  DoD 
laboratories,  test  centers,  universities,  and  industrial  sites.  Because  the  HPCMP  is  managed  as  a 
DoD-wide  corporate  asset,  expert  teams  are  guaranteed  access  to  a  world-class,  commercial, 

HPC  capability  based  on  mission  priority  rather  than  on  the  size  or  budget  of  their  organizations. 

A  User-Centered  Management  Focus 

Traditional  HPC  programs  are  built  around  a  single  central  facility  that  acts  as  the  “center  of 
gravity”  for  all  activities.  The  HPCMR  in  contrast,  provides  a  world-class  HPC  capability 
accessible  over  a  single  network  service,  the  DREN.  All  the  program’s  MSRCs  and  DCs  are 
available  to  HPCMP  users,  with  the  allocation  of  resources  determined  independently  by  the 
Services  and  Defense  Agencies.  Additionally,  users  have  a  direct  role  in  overall  policy  and 
implementation  through  the  HPC  User  Advocacy  Group  (UAG)  and  through  their  CTA  leaders, 
who  unify  the  HPCMP  community. 


Integrated  Program  Strategy 


section. 

The  HPCMP  continually 
acquires  the  best  commercially 
available,  high-end,  high 
performance  computers.  The 
program  also  acquires  and 
develops  common-use  software 
tools  and  programming 
environments,  expands  and 
trains  the  DoD  HPC  user  base, 
and  exploits  the  best  ideas, 
algorithms,  and  software  tools 
emerging  from  the  national  HPC 
infrastructure. 


Control 


and  Engineering 


The  program  has  evolved  beyond  an  initial  phase  of  procurement  and  program  development  to 
field,  in  FY  1996,  a  world-class  HPC  infrastructure  for  the  DoD  S&T  and  the  T&E  communities. 
Figure  1  depicts  the  HPCMP  integrated  program  strategy.  The  infrastructure  is  comprised  of  three 
components:  HPC  Centers, 

Networking,  and  Software 

Applications  Support.  Each  of  m 

these  components  will  be  °°‘rs 

discussed  in  detail  in  this  Mm  The 

72ls*CeoWrv 


Figure  1 .  Integrated  Program  Strategy  of  the  Department  of  Defense 
High  Performance  Computing  Modernization  Program 
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User  Requirements 

The  requirements  process,  developed  and  employed  by  the  program,  includes  an  annual  survey 
of  the  DoD  S&T  and  T&E  communities,  coupled  with  site  visits  to  discuss  requirements  and 
provide  information  to  user  groups.  The  program  obtains  additional  feedback  through  user 
satisfaction  surveys,  annual  user  group  conferences,  and  advisory  panels  who  establish  policies 
and  procedures  at  the  HPC  centers. 

HPC  systems  performance  metrics  are  tracked  throughout  the  year  at  the  computational  project 
level  to  ensure  projects  receiving  allocations  are  able  to  use  the  resources  effectively  and 
efficiently.  This  process  provides  feedback  to  the  Services  and  Agencies  before  allocating 
resources  the  following  fiscal  year.  The  overall  integrated  process  furnishes  accurate  and  timely 
data  on  which  to  base  program  decisions. 

World-Class  Shared  HPC  Capability 

The  HPCMP  provides  almost  27  teraflops  of  peak  computing  capability,  an  integrated  computing 
environment,  and  a  full  complement  of  specialized  and  general  user  support.  The  program 
provides  a  variety  of  architectures  and  classes  of  systems,  ranging  from  the  most  powerful 
commercial  configurations  available  from  major  US  HPC  vendors  to  more  specialized  systems 
configured  for  specific  types  of  mission  applications.  HPC  capability  is  upgraded  regularly  to 
ensure  that  DoD  users  are  among  the  first  to  use  the  new  generations  of  HPC  capabilities.  During 
FY  2002,  the  HPCMP  completed  its  third  technology  upgrade  at  the  MSRCs  and  continued  to  fund 
special  capabilities  at  selected  DCs.  The  program  will  continue  to  make  major  upgrades  to 
capabilities  at  the  MSRCs  in  FY  2003. 

DoD  HPC  Challenge  Projects 

HPCMP  allocates  approximately  25  percent  of  its  HPC  resources  each  fiscal  year  to  competitively 
selected  DoD  Challenge  Projects.  These  high-priority,  computationally  intensive  projects  develop 
and  use  pathfinding  applications  that  meet  mission-critical  warfighting  and  science  and 
technology  requirements.  Forty-one  projects  were  selected  for  implementation  in  fiscal  year  2002. 
Many  of  the  projects  use  multiple  hardware  platforms  and  in  some  cases,  multiple  shared 
resource  centers.  Table  2  lists  the  fiscal  year  2002  DoD  HPC  Challenge  Projects. 

HPCMP  Computational  Technology  Areas 

The  DoD  HPCMP  user  community  is  organized  around  ten  broad  CTAs.  Each  CTA  has  a 
designated  leader,  a  prominent  DoD  scientist  or  engineer,  working  in  the  discipline.  Users  of 
HPCMP  resources  address  challenges  in  the  following  computational  areas: 

•  Computational  Structural  Mechanics,  Mr.  Michael  Giltrud,  Defense  Threat  Reduction  Agency, 
Alexandria,  VA 

•  Computational  Fluid  Dynamics,  Dr.  Robert  Meakin,  Army  Aviation  and  Missile  Command, 
Moffett  Field,  CA 

•  Computational  Chemistry  and  Materials  Science,  Dr.  Ruth  Pachter,  Air  Force  Research 
Laboratory,  Materials/Manufacturing  Directorate,  Wright-Patterson  AFB,  OH 
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Table  2.  DoD  High  Performance  Computing  Challenge  Projects  (Fiscal  Year  2002) 


Project 

Project  Leader(s) 

Organization 

Computational  Structural  Mechanics 

Evaluation  and  Retrofit  for  Blast  Protection  in 

Urban  Terrain 

James  Baylot,  Tommy 

Bevins,  James  O'Daniel, 

Young  Shon,  David  Littlefield, 
and  Chrisopher  Eamon 

Engineer  Research  and 
Development  Center,  Vicksburg, 

MS 

Modeling  Complex  Projectile  Target  Interactions 

Kent  Kimsey  and  David 

Kleponis 

Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 

Three-Dimensional  Modeling  and  Simulation  of 
Bomb  Effects  for  Obstacle  Clearance 

Alexandra  Landsberg 

Naval  Surface  Warfare  Center, 

Indian  Head  Division,  Indian 

Head,  MD 

Computational  Fluid  Dynamics 

3-D  Computational  Fluid  Dynamics  Modeling  of 
the  Chemical  Oxygen-Iodine  Laser  (COIL) 

Timothy  Madden 

Air  Force  Research  Laboratory, 
Kirtland  AFB,  NM 

Active  Control  of  Fuel  Injectors  in  Full  Gas 

Turbine  Engines 

Suresh  Menon 

Army  Research  Office,  Research 
Triangle  Park,  NC 

Airdrop  System  Modeling  for  the  21st  Century 
Airborne  Warrior 

Keith  Stein,  Richard  Benney, 
Tayfun  Tezduyar,  James 
Barbour,  Hamid  Johari,  and 
Sonya  Smith 

Army  Soldier  &  Biological 

Chemical  Command,  Natick,  MA 

Analysis  of  Full  Aircraft  with  Massive  Separation 
Using  Detached-Eddy  Simulation 

Kenneth  Wurtzler,  Robert 
Tomaro,  William  Strang, 
Matthew  Grismer,  Capt  James 
Forsythe,  Kyle  Squires,  and 
Philiipe  Spalart 

Air  Force  Research  Laboratory, 
Wright-Patterson  AFB,  OH 

Applied  CFD  in  Support  of  Aircraft-Store 
Compatibility  and  Weapons  Integration 

John  Martel 

Air  Force  Seek  Eagle  Office, 

Eglin  AFB,  FL 

High-Fidelity  Analysis  of  UAVs  Using  Nonlinear 
Fluid/Structure  Simulations 

Reid  Melville  and  Miguel 

Visbal 

Air  Force  Research  Laboratory, 
Wright-Patterson  AFB,  OH 

High-Resolution  Adaptive  Mesh  Refinement 
(AMR)  Simulation  of  Confined  Explosions 

John  Bell,  Marcus  Day, 

Charles  Rendleman,  Mike 
Lijewski,  Vincent  Beckner, 
and  Allen  Kuhl 

Defense  Threat  Reduction 

Agency,  Alexandria,  VA 

Hybrid  Particle  Simulations  of  High  Altitude 

Nuclear  Explosions  in  3-D 

Stephen  Brecht 

Defense  Threat  Reduction 

Agency,  Alexandria,  VA 

Large-Eddy  Simulation  of  Steep  Breaking  Waves 
and  Thin  Spray  Sheets  Around  a  Ship:  The  Last 
Frontier  in  Computational  Ship  Hydrodynamics 

Ki-Han  Kim,  Dick  Yue,  and 
Douglas  Dommermuth 

Naval  Surface  Warfare  Center, 
Carderock  Division,  Bethesda, 

MD 

Large-Eddy  Simulation  of  Tip-Clearance  Flow  in 
Stator-Rotor  Combinations 

Parviz  Moin,  Meng  Wang, 
Donqhyun  You,  and  Rajat 

Mittal 

Office  of  Naval  Research, 

Arlington,  VA 

Modeling  of  Microbubble  Drag  Reduction 

Physics 

Parney  Albright  and  James 
Uhlman 

Defense  Advanced  Research 

Project  Agency,  Arlington,  VA 

Multiphase  CFD  Simulations  of  Solid  Propellant 
Combustion  in  Modular  Charges  and 
Electrothermal-Chemical  (ETC)  Guns 

Michael  Nusca  and  Paul 

Conroy 

Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 
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Table  2 — Continued.  DoD  High  Performance  Computing  Challenge  Projects 


Project 

Project  Leader(s) 

Organization 

Computational  Fluid  Dynamics  (continued) 

Numerical  Modeling  of  Wake  Turbulence  for 

Naval  Applications:  Vortex  Dynamics  and  Late- 
Wake  Turbulence  in  Stratification  and  Shear 

Michael  Gourlay,  Donald 

Delisi,  David  Fritts,  Maria- 
Pascale  LeLong,  James 

Riley,  Robert  Robins,  and 

Joe  Werne 

Office  of  Naval  Research, 

Arlington,  VA 

Parallel  Simulations  of  Weapons/Target 

Interactions  Using  a  Coupled  CFD/Computational 
Structural  Dynamics  (CSD)  Methodology 

Joseph  Baum 

Defense  Threat  Reduction 

Agency,  Alexandria,  VA 

Three-Dimensional,  Unsteady,  Multi-Phase  CFD 
Analysis  of  Maneuvering  High  Speed 
Supercavitating  Vehicles 

Robert  Kunz,  Jules  Lindau, 
and  Howard  Gibeling 

Office  of  Naval  Research, 

Arlington,  VA 

Time-Accurate  Aerodynamics  Modeling  of 

Synthetic  Jets  for  Projectile  Control 

Jubaraj  Sahu,  Sukumar 
Chakravarthy,  and  Sally  Viken 

Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 

Time  Accurate  Computational  Simulations  of 

Ship  Airwake  for  Dl,  Simulation  and  Design 
Applications 

Susan  Polsky 

Naval  Air  Warface  Center  Aircraft 
Division,  Patuxent  River,  MD 

Unsteady  RANS  Simulation  for  Surface  Ship 
Maneuvering  and  Seakeeping 

Ki-Han  Kim,  Joseph  Gorski, 

Fred  Stern,  Lafayette  Taylor, 
and  Mark  Hyman 

Naval  Surface  Warfare  Center, 
Carderock  Division,  Bethesda, 

MD 

Computational  Chemistry  and  Materials  Science 

Characterization  of  DoD  Relevant  Materials  and 
Interfaces 

Emily  Carter 

Air  Force  Office  of  Scientific 
Research,  Bolling  AFB,  DC 

Computational  Chemistry  Models  Leading  to 
Mediation  of  Gun  Tube  Erosion 

Cary  Chabalowski,  Margaret 
Hurley,  Dan  Sorescu,  Roger 
Ellis,  Donald  Thompson, 

Betsy  Rice,  and  Gerald 
Lushington 

Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 

First  Principles  Studies  of  Technologically 

Important  Smart  Materials 

Andrew  M.  Rappe 

Office  of  Naval  Research, 

Arlington,  VA 

Interactions  of  Chemical  Warfare  Agents  with 
Acetlychol  inesterase 

Jeffery  Wright,  Gerald 
Lushington,  and  William 

White 

Edgewood  Chemical  and 

Biological  Center,  Aberdeen 

Proving  Ground,  MD 

Multiscale  Simulations  of  High  Temperature 
Ceramic  Materials 

Rajiv  Kalia,  Aiichiro  Nakano, 
and  Priya  Vashista 

Air  Force  Office  of  Scientific 
Research,  Bolling  AFB,  DC 

Multiscale  Simulation  of  Nanotubes  and 

Quantum  Structures 

Jerry  Bernholc 

Office  of  Naval  Research, 

Arlington,  VA 

Multi-Scale  Simulations  of  High  Energy  Density 
Materials 

Jerry  Boatz 

Air  Force  Research  Laboratory, 
Edwards  AFB,  CA 

New  Materials  Design 

Jerry  Boatz,  Ruth  Pachter, 

Mark  Gordon,  Gregory  Voth, 
and  Sharon  Hammes-Schiffer 

Air  Force  Office  of  Scientific 
Research,  Bolling  AFB,  DC 
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Table  2 — Continued.  DoD  High  Performance  Computing  Challenge  Projects 


Project  Project  Leader(s)  Organization 


Computational  Electromagnetics  and  Acoustics 

Airborne  Laser  Challenge  Project  II 

Wilbur  Brown,  Jeremy  Winick, 
and  Robert  Beland 

Air  Force  Research  Laboratory, 
Kirtland  AFB,  NM 

Directed  High  Power  RF  Energy:  Foundation  of 
Next  Generation  Air  Force  Weapons 

Keith  Cartwright 

Air  Force  Research  Laboratory, 
Kirtland  AFB,  NM 

Radar  Signature  Database  for  Low  Observable 
Engine  Duct  Design 

Kueichien  Hill 

Air  Force  Research  Laboratory, 
Wright-Patterson  AFB,  OH 

Seismic  Signature  Simulations  for  Tactical 

Ground  Sensor  Systems  and  Underground 

Facilities 

Mark  Moran,  Stephen 

Ketcham,  George  McMechan, 
Stig  Hestholm,  Roy 

Greenfield,  Thomas 

Anderson,  James  Lacombe, 
Michael  Shatz,  Robert  Haupt, 
Richard  Lacoss,  and  Robert 
Greaves 

Cold  Regions  Research  and 
Engineering  Laboratory,  Hanover 

NH 

Signature  Modeling  for  Future  Combat  Systems 

Raju  Namburu,  Theresa 

Gonda,  Peter  Chung, 

Margaret  Hurley,  Steve  Bunte, 
William  Coburn,  Christopher 
Kenyon,  Calvin  Lee,  John 
Escarsega,  and  William 
Spurgeon 

Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 

Climate,  Weather,  and  Ocean  Modeling  and  Simulation 

1/32  Degree  Global  Ocean  Modeling  and 

Prediction 

Alan  Wallcraft,  Harley 

Hurlburt,  Robert  Rhodes,  and 
Jay  Shriver 

Naval  Research  Laboratory,  John 

C.  Stennis  Space  Center,  MS 

Basin-scale  Prediction  with  the  HYbrid 

Coordinate  Ocean  Model 

Eric  Chassignet 

Office  of  Naval  Research, 

Arlington,  VA 

Coupled  Environmental  Model  Prediction  (CEMP) 

Wieslaw  Maslowski,  Julie 
McClean,  Albert  Semtner, 

Robin  Tokmakian,  Yuxia 

Zhang,  Ruth  Preller,  and 

Steve  Piacsek 

Naval  Postgraduate  School  and 
Naval  Research  Laboratory, 
Monterey,  CA 

Coupled  Mesoscale  Modeling  of  the  Atmosphere 
and  Ocean 

Richard  Hodur 

Naval  Research  Laboratory, 
Monterey,  CA 

High  Fidelity  Simulation  of  Littoral  Environments 

Richard  Allard  and  Jeffery 
Holland 

Naval  Research  Laboratory, 

John  C.  Stennis  Space  Center, 

MS  and  Engineer  Research  and 
Development  Center,  Vicksburg, 

MS 

Submerged  Wakes  in  Littoral  Regions 

Patrick  Purtell,  Roger  Briley, 
Joseph  Gorski,  and  Douglas 
Dommermuth 

Office  of  Naval  Research, 

Arlington,  VA 

Signal/lmage  Processing 

Automatic  Target  Recognition  Performance  Timothy  Ross  Air  Force  Research  Laboratory, 

Evaluation  Wright-Patterson  AFB,  OH 
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•  Computational  Electromagnetics  and  Acoustics,  Dr.  Kueichien  Hill,  Air  Force  Research 
Laboratory,  Sensors  Directorate,  Wright-Patterson  AFB,  OH 

•  Climate/Weather/Ocean  Modeling  and  Simulation,  Dr.  William  Burnett,  Naval  Oceanographic 
Office,  John  C.  Stennis  Space  Center,  MS 

•  Signal/lmage  Processing,  Dr.  Anders  Sullivan,  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD 

•  Forces  Modeling  and  Simulation/C4l,  Dr.  Larry  Peterson,  Space  and  Naval  Warfare  Systems 
Center,  San  Diego,  CA 

•  Environmental  Quality  Modeling  and  Simulation,  Dr.  Jeffrey  Holland,  US  Army  Corps  of 
Engineers,  Engineer  Research  and  Development  Center,  Vicksburg,  MS 

•  Computational  Electronics  and  Nanoelectronics,  Dr.  Barry  Perlman,  Army  Communications- 
Electronics  Command  Research  Development  and  Engineering  Center,  Ft.  Monmouth,  NJ 

•  Integrated  Modeling  and  Test  Environments,  Mr.  Jere  Matty,  Arnold  Engineering  Development 
Center,  Arnold  AFB,  TN 

Program  Components 


The  program  is  organized  into  three  components  to  achieve  its  goals.  The  components  are 
HPCMP  HPC  centers,  networking,  and  software  applications  support.  The  components  allow  the 
program  to  focus  on  the  most  efficient  means  of  supporting  the  S&T  and  T&E  communities’ 
requirements.  The  components  align  to  the  program  goals  as  follows: 

High  Performance  Computing  Centers 

The  High  Performance  Computing  Modernization  Program  currently  supports  four  MSRCs  and 
eleven  DCs.  The  large  MSRCs  provide  a  wide  range  of  high  performance  computing  (HPC) 
capabilities  designed  to  support  the  ongoing  needs  of  the  DoD  HPC  community. 

Each  MSRC  supports  complete  centralized  systems  and  services.  The  MSRCs  also  provide 
vector  machines,  scalable  parallel  systems,  clustered  workstations,  scientific  visualization 
resources,  and  access  to  training.  Storage  capabilities  include  distributed  file  and  robotic  archival 
systems.  The  four  MSRCs  are: 

•  Air  Force  Aeronautical  Systems  Center  (ASC),  Wright-Patterson  AFB,  OH 

•  Army  Research  Laboratory  (ARL),  Aberdeen  Proving  Ground,  MD 

•  Engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS 

•  Naval  Oceanographic  Office  (NAVO),  John  C.  Stennis  Space  Center,  MS 


The  DCs  provide  specialized  HPC  capability  to  users.  HPC  systems  have  been  established  at 
DCs  where  there  is  a  significant  advantage  to  having  local  HPC  systems  and  where  there  is  a 
potential  for  advancing  DoD  applications  using  investments  in  HPC  capabilities  and  resources 
(e.g.,  a  test  activity  with  real-time  processing  requirements). 
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The  DCs  are  divided  into  two  categories:  “Allocated”  and  “Dedicated”.  Allocated  Distributed 

Centers  associates  the  annual  service/agency  project  allocation  and/or  challenge  project 

allocation  process  with  these  centers.  Dedicated  Distributed  Centers  associates  the  dedicated 

project  or  projects  for  which  each  independent  DDC  was  awarded  with  their  respective  center. 

The  DCs  have  the  following  goals: 

•  Apply  commercial  HPC  hardware  and  software  as  rapidly  as  it  becomes  available. 

•  Complement,  balance,  and  supplement  the  major  shared  resource  centers  by  enabling  local 
expertise  to  be  developed  and  leveraged  by  the  larger  DoD  community. 

•  Support  small  and  medium-size  HPC  applications,  leveraging  the  larger  MSRCs  that  support 
large  applications. 

•  Foster  reuse  of  software  tools  and  application  software  components  as  well  as  appropriate  use 
of  communication  standards,  interface  standards,  and  graphics  visualization  standards  across 
DoD. 

•  Support  local  HPC  requirements  at  selected  sites  where  there  is  potential  for  advancing  DoD 
applications. 

•  Promote  the  development  of  new  software  tools  and  application  area  specific  software. 

•  Leverage  HPC  expertise  and  assets  located  in  industry,  academia  and  other  federal 
laboratories  in  addition  to  DoD  facilities. 


The  DCs  are  : 

Dedicated  Distributed  Centers 

•  Air  Force  Research  Laboratory,  Information  Directorate  (AFRL/IF),  Rome,  NY 

•  Arnold  Engineering  Development  Center  (AEDC),  Arnold  AFB,  TN 

•  Fleet  Numerical  Meteorology  and  Oceanography  Center  (FNMOC),  Monterey,  CA 

•  Naval  Air  Warfare  Center  Aircraft  Division  (NAWCAD),  Patuxent  River,  MD 

•  Naval  Research  Laboratory  (NRL-DC),  Washington,  DC 

•  Simulation  and  Analysis  Facility  (SIMAF),  Wright-Patterson  AFB,  OH 

•  White  Sands  Missile  Range  (WSMR),  White  Sands  Missile  Range,  NM 
Allocated  Distributed  Centers 

•  Arctic  Region  Supercomputing  Center  (ARSC),  Fairbanks,  AK 

•  Army  High  Performance  Computing  Research  Center  (AHPCRC),  Minneapolis,  MN 

•  Maui  High  Performance  Computing  Center  (MHPCC),  Kihei,  HI 

•  Space  and  Missile  Defense  Command  (SMDC),  Huntsville,  AL 

The  MSRCs  and  DCs  provide  state-of-the-art  HPC  capabilities  operating  in  several  classification 
environments  and  ensuring  the  DoD  science  and  technology  and  test  and  evaluation  user 
communities  have  the  computational  tools  needed  to  retain  the  US  warfighter’s  technological 
advantage. 
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Figure  2.  DREN  Connections,  High  Performance  Computing  Centers,  and  Other  Network  Access  Points 


Networking 

Defense  Research  and  Engineering  Network 

The  DREN  is  DoD’s  recognized  research  and  engineering  network.  The  DREN  is  a  robust,  high 
bandwidth,  low  latency  network  that  provides  connectivity  between  the  High  Performance 
Computing  Program’s  geographically  dispersed  user  sites  and  high  performance  computing 
centers.  (See  Figure  2) 

DREN  networking  services  are  provided  as  a  Virtual  Private  Network  over  the  public  infrastructure. 
DREN  currently  supports  both  cell-based  and  packet-based  traffic  over  core  infrastructures  such 
as  Multi  Protocol  Layer  Service  and  ATM.  DREN  Service  Delivery  Points  are  specified  in  terms  of 
WAN  bandwidth  access,  network  protocols,  and  local  interfaces  for  connectivity.  Minimal  access 
to  a  site  is  DS-3  (45  Mbps)  with  potential  high  end  access  of  OC-768  (40  Gbps)  over  the  next  10 
years.  Current  site  connectivity  to  the  DREN  ranges  from  DS-3  at  some  user  sites  to  OC-1 2  (622 
Mbps)  at  MSRCs  and  selected  DCs  and  Internet  Network  Access  Points  (NAP). 

Since  its  inception,  DREN  has  been  very  active  in  transferring  leading  edge  network  and  security 
technologies  across  DoD  and  other  Federal  agencies.  Our  leading  edge,  defense-in-depth 
security  model  is  now  viewed  by  experts  as  the  best  way  to  protect  resources  both  internally  and 
externally.  Layered  defense  extends  from  access  control  lists  at  the  NAPs  through  sophisticated, 
high  bandwidth  intrusion  detection  monitored  by  our  HPC  Computer  Emergency  Response  Team 
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Hardware  pre-authentication,  strong  configuration  management,  and  periodic  physical  security 
assessments  all  contribute  to  hardened  resources  that  are  still  accessible  to  the  DoD  research 
scientists  from  wherever  they  need  to  work.  Well  designed  WAN  security  coupled  with  strong 
server  and  user  security,  allows  DREN  to  truly  deploy  leading-edge,  defense-in-depth  protection  of 
DoD  resources. 

DREN  services  are  currently  provided  to  sites  throughout  the  continental  United  States,  Alaska, 
Hawaii,  and  can  be  extended  overseas  where  necessary.  Incorporating  the  best  operational 
capabilities  of  both  the  DoD  and  the  commercial  telecommunications  infrastructure,  DREN 
provides  DoD  a  secure,  world-class,  long-haul  research  network  in  support  of  DoD’s  S&T  and 
T&E  communities.  DREN  also  extends  services  to  DoD’s  Modeling  &  Simulation  community  and 
the  MDA. 

Security 

The  HPCMP  has  also  designed  and  deployed  a  Secret  DREN  (SDREN)  using  a  common  Secret 
systems  high  key  with  National  Security  Agency  certified  Type-1  encryptors.  SDREN  can 
transport  classified  traffic  at  OC-3  (155  Mbps). 

The  Defense  Information  Systems  Agency  Information  Security  Program  Management  Office  has 
certified  the  DREN  for  sensitive  but  unclassified  information  and  encrypted  classified  traffic. 

Access  to  high  performance  computing  systems  is  controlled  by  enhanced  security  technology 
and  user  procedures. 

Software  Applications  Support 

“Software  Applications  Support”  is  new  terminology  that  captures  the  evolutionary  nature  of  the 
Program’s  efforts  to  acquire  and  develop  joint  need  HPC  applications,  software  tools,  and 
programming  environments  and  educate  and  train  DoD’s  scientists  and  engineers  to  effectively 
use  advanced  computational  environments.  This  component  contains  two  areas:  the  CHSSI, 
which  has  been  producing  efficient  applications  software  codes  for  the  last  several  years  and  PET, 
which  is  realigned  from  the  HPC  MSRCs.  As  the  program  has  evolved,  the  potential  synergy  of  re¬ 
focusing  PET  activities  caused  the  HPCMP  to  realign  PET  from  a  shared  resource  center-centric 
orientation  to  a  broader  program-wide  focus.  This  realignment  offers  significant  opportunities  for 
leveraging  and  collaborating  with  CHSSI. 

Common  High  Performance  Computing  Software  Support  Initiative 

CHSSI  consists  of  a  set  of  focused  software  development  activities.  These  activities  are  designed 
to  provide  recognized  communities  of  DoD  computational  scientists  and  engineers  with  common 
HPC  technical  codes  that  exploit  scalable  computer  architectures.  Each  CHSSI  application  is 
selected  based  on  DoD  needs  and  technical  quality.  The  products  have  application  across  a 
significant  portion  of  the  DoD  S&T  and  T&E  HPC  user  community. 

CHSSI  provides  efficient,  scalable,  portable  software  codes,  algorithms,  tools,  models,  and 
simulations  that  run  on  a  variety  of  HPC  platforms  for  use  by  a  large  number  of  S&T  and  T&E 
scientists  and  engineers.  CHSSI,  which  is  organized  around  ten  CTAs  and  eight  portfolios, 
involves  many  scientists  and  engineers  working  in  close  collaboration  across  government, 
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industry,  and  academia.  CHSSI  teams  include  algorithm  developers,  applications  specialists, 
computational  scientists,  computer  scientists,  and  engineers. 

Currently,  the  CHSSI  program  invests  $20  million  per  fiscal  year  in  55  software  projects.  These 
support  a  mix  of  single  and  multi-disciplinary  projects.  These  projects  are  providing  “state-of-the- 
art”  or  advanced  computer  modeling  capabilities.  Prominent  scientists  and  engineers  from  the 
DoD  lead  the  projects.  These  development  efforts  leverage  the  considerable  expertise  and  assets 
of  industry,  academia,  and  other  federal  agencies.  CHSSI  software  products  are  rigorously  tested, 
documented,  and  reviewed  as  part  of  their  structured  development  process. 


Programming  Environment  and  Training 

PET  activity  focuses  on  improving  DoD  HPC  productivity  by  creating  an  information  conduit 
between  DoD  HPC  users  and  top  academic  and  industry  experts.  PET  is  responsible  for 
gathering  and  deploying  the  best  ideas,  algorithms,  and  software  tools  emerging  from  the  national 
high  performance  computing  infrastructure  into  the  DoD  user  community.  PET  acquires  world- 
class  technical  support,  develops  strategic  partnerships,  and  focuses  on  HPC  tool  development 
and  deployment. 

PET  is  divided  into  four  components  and  each  component  executes  a  program  that  supports  the 
designated  functional  area,  as  shown  in  Table  3. 


The  functional  areas  are 
grouped  to  encourage  synergy 
among  related  CTAs, 
collaboration  and  interaction 
between  CTA  and  cross¬ 
community  functions,  and  to 
balance  the  workload  across 
the  four  components. 

PET  activities  consist  of  short- 
and  long-term  strategic  efforts. 
Short-term  efforts  lasting  less 
than  three  years  include  short 
collaborative  projects  with 
users.  Two  of  the  long-term 
efforts  focus  on  distance 
learning  and  metacomputing. 
PET  teams  anticipate  and 
respond  to  the  needs  of  DoD 
users  for  training  and 
assistance.  The  teams  have  a 
strong  on-site  presence  at 
HPCMP  major  shared  resource 
centers. 


Table  3.  CTA  Component  Functional  Areas 


Component 

Number 

CTAs  and  Functional  Areas 

1 

Climate/Weather/Ocean  Modeling  and  Simulation 
Environmental  Quality  Modeling  and  Simulation 

Computational  Environment 

2 

Forces  Modeling  and  Simulation/C4l 

Integrated  Modeling  and  Test  Environments 

Signal/lmage  Processing 

Enabling  Technologies 

3 

Computational  Fluid  Dynamics 

Computational  Structural  Mechanics 

PET  Online  Knowledge  Center 

Education,  Outreach,  and  Training  Coordination 

4 

Computational  Chemistry  and  Materials  Science 
Computational  Electromagnetics  and  Acoustics 
Computational  Electronics  and  Nanoelectronics 

Collaborative  and  Distance  Learning  Technologies 
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1- D 

2- D 

3- D 

3DVAR 

AAA 

AATD 

AChE 

ADCIRC 

ADCP 

ADH 

ADMCPA 

ADVISR 

AFAP 

AFB 

AFOSR 

AFRL 

AFWA 

AHPCRC 

AIRMS 

AL 

ALE 

AMDSim 

AMR 

AoA 

API 

ARL 

ARO 

ASC 

ATD 

ATM 

ATR 

AWS 

BAD 

BLOS 

BMDS 

BN 

C 

CAD 

one-dimension  (dimensional) 
two-dimensions  (dimensional) 
three-dimensions  (dimensional) 

three-dimensional  variational  data  assimilation 

Anti-Aircraft  Artillery 

Aviation  Applied  Technology  Directorate 
acetylcholinesterase 

ADvanced  CIRCulation  Model 

Acoustic  Doppler  Current  Profiler 

ADvanced  Hydraulics 

2-azido-N,N-dimethylcyclopropanamine 

Advanced  Visualization  for  Intelligence,  Surveillance,  and  Reconnaissance 
as-fast-as-possible 

Air  Force  Base 

Air  Force  Office  of  Scientific  Research 

Air  Force  Research  Laboratory 

Air  Force  Weather  Agency 

Army  High  Performance  Computing  Research  Center 

Airborne  Infrared  Measurement  System 

aluminum 

Arbitrary  Lagrangian  Eulerian 

Air  and  Missile  Defense  Simulation 

Adaptive  Mesh  Refinement  or  Automatic  Mesh  Refinement 

Analysis  of  Alternatives  or  angle  of  attack 
applications  programming  interface 

Army  Research  Laboratory 

Army  Research  Office 

Aeronautical  Systems  Center 

Advanced  Technology  Demonstrator 

Asynchronous  Transfer  Mode 

Automatic  Target  Recognition 

Abrupt  Wing  Stall 

Behind-armor  debris 

beyond-line-of-sight 

Ballistic  Missile  Defense  System 

boron  nitride 

carbon 

computer  aided  design 
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C/B 

COM 

CEA 

CEM 

CEN 

CFD 

CHSSI 

CIO 

chemical/biological 

Computational  Chemistry  and  Materials  Science 

Computational  Electromagnetics  and  Acoustics 

Computational  Electromagnetics 

Computational  Electromagnetics  and  Nanoelectronics 

Computational  Fluid  Dynamics 

Common  High  Performance  Computing  Software  Support  Initiative 

Chief  Information  Officer 

COAMPS  Coupled  Ocean  Atmosphere  Mesoscale  Prediction  System 
CODESSA  Comprehensive  DEscriptors  for  Structural  and  Statistical  Analysis 
COMPOSE  Composite  Manufacturing  PrOcess  Simulation  Environment 


CONOPS 

CSD 

CSM 

CTA 

CWO 

DARPA 

DC 

DES 

DFT 

DMAZ 

DNS 

DoD 

DOF 

DON 

DREN 

DSD 

DTRA 

EQM 

ERDC 

ETC 

FCS 

FDTD 

FE 

FEM 

FeO 

FMM 

FMS 

FNC 

FNMOC 

FOX-7 

concept  of  operations 

Computational  Structural  Dynamics 

Computational  Structural  Mechanics 

Computational  Technology  Area 

Climate/Weather/Ocean  Modeling  and  Simulation 

Defense  Advanced  Research  Projects  Agency 

Distributed  Center 

Detached-Eddy  Simulation 
density  functional  theory 

2-azido-N,N-dimethylethanamine 

direct  numerical  simulation 

Department  of  Defense 

Degrees  of  Freedom 

Department  of  Navy 

Defense  Research  and  Engineering  Network 

Deforming-Spatial-Domain 

Defense  Threat  Reduction  Agency 

Environmental  Quality  Modeling 

US  Army  Corps  of  Engineers,  Engineer  Research  and  Development  Center 

electrothermal-chemical 

Future  Combat  Systems 

Finite  Difference  Time  Domain 

ferroelectric 

Finite  Element  Method 

iron  oxide 

fast  multipole  method 

Forces  Modeling  and  Simulation 

Future  Naval  Capabilities 

Fleet  Numerical  Meteorology  and  Oceanography  Center 

1 ,1  -diamino-2, 2-dinitroethylene 
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FSI 

FY 

GB 

GW 

HE 

HFM 

HFSOLE 

HLD 

HMX 

HPC 

HPCMP 

HPCMPO 

HPM 

HVRB 

HWILT 

HYCOM 

IBCTs 

ICE 

ICEPIC 

IMT 

I/O 

IP 

IRFNA 

JNIC 

JSF 

JTIDS 

KE 

LANL 

LB/TS 

LBE 

LES 

LO 

LOB 

M&S 

MACS 

MBPT(2) 

MCEL 

MCM 

MDA 

MDWAR 

MEMS 


fluid-structure  interaction 

fiscal  year 

gigabyte 

giga-watt 

high  explosives 

Hyperbolic  Frequency  Modulated 

High  Fidelity  Simulations  of  Littoral  Environments 

high-loading-density 

1 ,3,5,7-tetranitro-1 ,3,5,7-tetraazacyclooctane 

high  performance  computing  or  high  performance  computer 

High  Performance  Computing  Modernization  Program 

High  Performance  Computing  Modernization  Program  Office 

high  power  microwave 

history  variable  reactive  burn 

hardware-in-the-loop  test 

HYbrid  Coordinate  Ocean  Model 

Interim  Brigade  Combat  Teams 

Interdisciplinary  Computing  Environment 

Improved  Concurrent  Electromagnetic  Particle-in-Cell 

Integrated  Modeling  and  Test  Environment 

input/output 

Information  Processor  or  Internet  protocol 
inhibited  red  fuming  nitric  acid 
Joint  National  Integration  Center 
Joint  Strike  Fighter 

Joint  Tactical  Information  Distribution  System 
Kinetic  Energy 

Los  Alamos  National  Laboratory 

Large  Blast  and  Thermal  Simulator 

Lattice-Boltzman  Equation 

Large-Eddy  Simulations 

Low  observables/observability 

line-of-bearing 

Modeling  and  Simulation 

Modular  Artillery  Charge  System 

Many-body  perturbation  theory  (second  order) 

Model  Coupling  Environmental  Library 
mine  countermeasures 
Missile  Defense  Agency 

Missile  Defense  Wargame  and  Analysis  Resource 
micro-electro-mechanical  systems 
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METOC 

MGD 

MILO 

MLFMA 

MMH 

MoM 

MP2 

MPI 

MPRS 

MRTD 

MSRC 

MW 

NAP 

Meteorology  and  Oceanography  Community 
magnetogasdynamics 
magnetically  insulated  line  oscillator 
multi-level  fast  multipole  algorithm 
monomethylhydrazine 

method  of  moments 

Moller-Plesset  perturbation  theory  (second  order) 

Message  Passing  Interface 

Multi-Point  Refueling  System 

Multiresolution-Time  Domain 

Major  Shared  Resource  Center 
mega-watt 

Network  Access  Point 

NAVAIR  Naval  Air  Systems  Command 

NAVMETOCCOM  Naval  Meteorology  and  Oceanography  Command 


NAVO 

NAWCAD 

NE 

NGEN 

NIAB 

NISA 

NLOS 

NM 

NMR 

NOGAPS 

NRL 

Naval  Oceanographic  Office 

Naval  Air  Warfare  Center  Aircraft  Division 

nuclear  explosive 

The  Army’s  Next  Generation  Interior  Ballistics  Model 

Non-Ideal  Air  Blast 

Network  Infrastructure  Services  Agency 
non-line-of-sight 

nitromethane 

Nuclear  Magnetic  Resonance 

Navy  Operational  Global  Atmospheric  Prediction  System 
Naval  Research  Laboratory 

ns 

NWP 

ONR 

O-PDES 

PDF 

PenRen 

PET 

P-l 

POPS 

PVC 

PVP 

PZT 

QC 

QM/MM 

QSPR 

nanosecond 

Numerical  Weather  Prediction 

Office  of  Naval  Research 

Optimistic,  Parallel,  Discrete  Event  Simulation 
pair  distribution  function 

Pentagon  Renovation  Office 

Programming  Environment  and  Training 
pressure-impluse 

Primary  Oceanographic  Prediction  System 
permanent  virtual  circuit 
permanent  virtual  path 

lead  zirconate  titanate 

quantum-chemical 

quantum  mechanics/molecular  mechanics 

Quantitative  Structure-Property  Relationships 
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R&D 

RANS 

RCS 

RDT&E 

RF 

RHA 

RMS 

RPM 

ROC 

rt-PVC 

rt-PVP 

rt-VBR(s) 

SAFEDI 

SALVO  PS 

SBCCOM 

SDOF 

SDREN 

SGI 

SHAMRC 

SHARC 

SHMEM 

SIP 

SIPRNET 

SONAR 

SPAWAR 

SPE 

SPEEDES 

SSC 

SSCSD 

SSHB 

SST 

S&T 

STWAVE 

SVC 

SWAFS 

SWAN 

TARDEC 

TBC 

T&E 

Ti 

UAG 


research  and  development 
Reynolds  Averaged  Navier-Stokes 
radar  cross  section 

Research,  Development,  Test,  and  Evaluation 

radio  frequency 

rolled  homogenous  armor 

Root  Mean  Square 

rate  per  minute 

Receiver  Operating  Characteristic 
real-time  permanent  virtual  circuit 
real-time  permanent  virtual  path 
real-time  variable  bit  rate(s) 

Ship  Aircraft  Airwake  Analysis  for  Enhanced  Dynamic  Interface 
Salvage  Operation (s) 

Soldier  Biological  and  Chemical  Command 
single-degree-of-freedom 

Secret  Defense  Research  and  Engineering  Network 
Silicon  Graphics  Inc. 

Second-order  Hydrodynamic  Automatic  Mesh  Refinement  Code 

Second-order  Hydrodynamic  Advanced  Research  Code 

SHared  MEMory 

Signal/Image  Processing 

Secret  Internet  Protocol  Routed  Network 

SOund  Navigation  and  Ranging 

Space  and  Naval  Warfare  Systems  Command 

Scalable  Processing  Environment 

Synchronous  Parallel  Environment  for  Emulation  and  Discrete-Event  Simulation 

SPAWAR  Systems  Center 

SPAWAR  Systems  Center,  San  Diego 

Short  Strong  Hydrogen  Bond 

Stabilized  Space-Time 

science  and  technology 

STeady-state  WAVE 

Scientific  Visualization  Center 

Shallow  Water  Analysis  and  Forecast  System 

Simulating  WAves  Nearshore 

Tank  Automotive  Research,  Development  and  Engineering  Center 
Thermal  Barrier  Coating 
test  and  evaluation 
titanium 

User  Advocacy  Group 
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UAV 

Unmanned  Air  Vehicles 

UDMH 

unsymmetrical  dimethylhydrazine 

UGS 

Unattended  Ground  Sensor 

UHF 

Ultra  High  Frequency 

UNCLE 

Unsteady  Computations  of  Field  Equations 

U.S. 

United  States 

UWB 

ultra-wideband 

UXO 

Unexploded  Ordnance 

VASP 

Vienna  as  Initio  Simulation  Package 

VCMD 

variable-charge  molecular  dynamics 

VDG 

Virtual  Data  Grid 

VHF 

very  high  frequencies 

W 

Vertical  Transmit-Vertical  Receive 

WAN 

wide  area  network 

WG2000 

Wargame  2000 

WLS 

weighted,  least-squares 

WRF 

Weather  Research  and  Forecast 

WSMR 

White  Sands  Missile  Range 

YSZ 

yttria-stabilized  zirconia 

Zr 

zirconate 
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